The Evaluation of a Scientific Magazine

One of the main activities of a researcher is the publication of articles related to their research topics.

The evaluation of the quality of research is essential to access the funding that can come, for example, through participation in European Projects. When public funding is obtained, it is necessary that the results of the research are then available to everyone free of charge and immediately. A good Research Quality Assessment is also essential in order to undertake the academic career.

Therefore, in evaluating a Journal today it is very important that this journal is in Open Access, other fundamental characteristics are a regular periodicity, the ISSN code, the translation of at least the abstract into English, the presence of keywords and the presence of a relevant Scientific Committee, possibly international. The journals are mainly divided into two large groups: the scientific ones and the humanistic ones. The former are characterized by the Impact Factor, that is, an index resulting from a relationship between the citations of articles published in the previous two-year period and the total number of articles published in that same two-year period by the magazine. The citations of the articles flow into the Journal Citation Report, Scopus and the Web of Science:

• Science Citation Index Expanded covers more than 8,500 noteworthy journals spanning 150 disciplines. Coverage goes from the year 1900 to the present day.
• The Social Science Citation Index covers more than 3,000 journals in the disciplines of the social sciences. The range of coverage goes from the year 1900 to the present day.
• Arts & Humanities Citation Index covers more than 1,700 arts and humanities journals since 1975. In addition, 250 leading scientific and social science journals are also covered.
• Emerging Sources Citation Index covers over 5,000 science, social science and humanities journals.
• Book Citation Index covers over 60,000 books selected in an editorial fashion since 2005.
• Conference Proceedings Citation Index (CPCI) covers more than 160,000 science conference titles from 1990 to present.

For example, calculating IF 2018 for the journal under consideration:

75 2018
Citations of articles published in 2016-17 75 = 0.484
155 Total articles published in 2016-17

The journals of the humanistic type are instead organized into classes: in class A we find the most prestigious journals for each discipline, journals that have undergone a peer review process, which come out regularly, already present for several years, with abstracts in English etc.; to ac- cess Class A, a very documented request must be made.

An interesting article by Faggiolani and Solimine on the evaluation issue was published ten years ago on AIB Studi¹. If we search for it on the page of Aib Studi, we can see various things: on the right where this magazine is indexed (ANVUR, DOAJ, SCOPUS), in the center the DOI, the abstract, all the important data, a graph with the downloads updated up to last month, the references and, lastly, how the article should be cited.

The article highlights that, for publications in the humanities, the monograph is often more significant than the single article and publication in a language other than English is penalized. Often the publications in the humanities are not in multiple hands and this greatly reduces the number of citations. Furthermore, for humanistic journals, the scientific value generally extends for a much longer time, even for many years. The importance of peer review is therefore upheld, although there are also problems in this practice. So perhaps the best solution would be a mixed approach that takes into account some “measurable” factors but also double-blind peer review (mutually unknown authors and evaluators). The article imagines that libraries could give some impetus to under-standing the real impact of a publication on the rest of the scientific world, evaluating the existence of remains and the presence in catalogs of prestigious libraries, with methods such as the LCA (Library Catalog Analysis). Other authors, such as Figà Talamanca, launched themselves, twenty years ago, against the use of IF in mathematics, because “friends” quotes were also taken into account, or else “negative” quotes were included in the count. Ultimately, it is not certain that a large quantity necessarily also means quality².

Fig. 1. C. Faggiolani e G. Solimine, 2012, La valutazione della ricerca, la bibliometria e l’albero di Bertoldo, AIB studi, 52, 1 (mag. 2012). DOI: https://doi.org/10.2426/aibstu- di-6290.

According to Antonella De Robbio: “For the purpose of measuring the impact of an author’s work within the scientific community, it is essential to start experimenting with new bibliometric techniques within the open archives together with the development of new” open “modalities aimed at satisfying the sophisticated needs for a correct evaluation of intellectual research productions. […] Open Access opens up new frontiers, not only in the development of new generation bibliometric indicators focused on the single work or in any case on the author rather than on the periodical, but above all in terms of infrastructure useful for wide-ranging impact assessment”³. De Robbio also cites Daniel Torres-Salinas’s 2009 study on LCA for monographs, which is the most popular product for human sciences⁴.

In 2014, Simona Turbanti⁵ in Navigating the sea by Scop- us Web Of Science and Google Scholar describes the research – carried out in the two large citation databases, Web of science and Scopus, and in Google Scholar – of the scientific works of the professors and researchers of the M-STO sector / 08, archives and librarianship, illustrating the method followed, the main differences in use as well as the limitations of the queried databases. WoS searches in over 12,300 journals, about 150,000 conference proceedings and 30,000 monographic publications, with a clear prevalence of titles from the North American area, while the ‘younger’ Scopus covers about 19,500 periodical titles, many of which from Europe, over 350 collections of monographs (and, starting from 2013, also single monographs), almost 5,000,000 conference papers and patents.

By consulting the Journal Citation Report, an annual report produced by the ISI (Institute for Scientific Information), where you can find the statistics on the number of citations that are made within a large number of technical-scientific journals, we can find, besides the Impact Factor, other indicators that allow us to understand the value of a magazine:

• Immediacy Index: measures how quickly an article from a journal is cited on average and how often the articles from that journal are cited in the current
• Cited Half-Life: measures the average duration of citations of articles in a magazine in the current year, or rather the ongoing relevance of a publication.
• Rate of Cites Index: a quality index of the single article, the more the work is cited, the more significant its scientific value.

Citation Impact: results from the ratio between the number of citations and the number of published works. It directly refers to the usefulness that the publications of a nation, university, research structure or even of a single researcher have had in the scientific world.

For example, looking for the Scientific Journal in the JCR Coastal Engineering Journal we find the description of the journal, place of publication, periodicity, IF, with this brief description in English of the IFJ: “it is a journal-level metric calculated from data indexed in the Web of Science Core Collection. It should be used with careful attention to the many factors that influence citation rates, such as the volume of publication and citations characteristics of the subject area and type of journal. The Journal Impact Factor can complement expert opinion and informed peer review. In the case of academic evaluation for tenure, it is inappropriate to use a journal-level metric as a proxy measure for individual researchers, institutions, or articles”.

In addition to the numerical data, for example, the IF of 2020 is 3216 (very high) and by eliminating self-citations, it is 2811, that is still very high. It is interesting to also see the trend of the IF in a graphic representation.

The classification of humanistic journals is an activity carried out by ANVUR (National Agency for the Evaluation of the University and Research System) for the purpose of calculating the indicators of the National Scientific Qualification starting from 2012. It is also needed for the purposes of accrediting PhD courses, in relation to requirement A4.3 starting from the XXXIII cycle (a.y. 2017-18), carried out exclusively for the sectors envisaged by the relevant legislation, that is, those that are part of the human and social sciences and identified as “non-bibliometric”. The classification is divided into 6 Areas:

• Architecture (Area CUN / VQR 8.a).
• Ancient, philological-liter- ary and historical-artistic sciences (Area 10).
• Historical, philosophical and pedagogical sciences (Area 11.a).
• Legal Sciences (Area 12).
• Economic and statistical sciences (Area 13).
• Political and social sciences (Area 14)⁶.

According to the ANVUR regulation, in order to be included in the list of class A scientific publications, a journal must ensure ‘double blind’ refereeing procedures (double-blind review). On the contrary, the evaluation of research products through the VQR is only ‘one-side blind review’ because the evaluator is aware of the identity of the author. This lack of anonymity can lead to evaluating the author rather than the work.

Therefore, before deciding where to publish, a shrewd researcher chooses a Journal that has a good Impact Factor or is in Class A.

What happens instead for multidisciplinary journals, as in the case of our UniCamillus Global Heath Journal? That is, that they cannot be inserted within these two categories?

On the ANVUR website on 15/02/2016, therefore now 6 years ago, it is specified as follows: “Clarification on the treatment of multidisciplinary journals (such as Nature, Science, etc.) in bibliometric evaluation. The articles published in these journals will obviously be accepted for evaluation, and can be evaluated with the bibliometric method. As already indicated in the accompanying document to the publication of the Web of Science data and in the clarification note on the application of the bibliometric algorithm, in fact, the products published in these journals, during the bibliometric evaluation, will be evaluated using the thresholds of the SC a to which the majority of the articles cited by the product in the bibliography belong and those that have cited the product itself”.

Therefore, the majority of the articles published in the journal will be considered for evaluation purposes.

An interesting little volume was recently published in the USA: Zaumanis M. (2021), Write an impactful research paper: A scientific writing technique that will shape your academic career, ISBN 13 9798680546949.

The author is a young PhD scholar in the USA, currently a researcher in Switzerland at Empa (a research group that is part of ETH).

The book aims to show strategies and tools in order to direct aspiring academics and authors toward the good writing of successful scientific articles. It is aimed at readers specialized in any scientific and humanistic field. The intention is to reach above all young researchers, even (but perhaps above all) without publications. The attempt to stimulate this type of audience masks the subtle criticism of the multidisciplinary academic world, at times so elitist and frightening as to seem unattainable. The text is divided into two parts.

Part 1: The Leap Writing Approach

EAP academic writing approach is a schematic and systematic approach to the paper and an indication for selecting the right periodical in which to publish it. From a linguistic point of view, reference is made to both the form and the vehicular language of the research. The importance and effectiveness of a scientific article reside in the message you want to communicate to the reader, not in the selection of high-sounding and superfluous terms that could only confuse. As for the choice of English as a vehicular language, it is undoubtedly the most sensible choice to reach a wider audience, nevertheless it is not necessary to have a high level of competence and command of it. As for the contents, every academic work deserves attention, as well as the idea behind it and the results obtained.

The LEAP principle

L: Layout, graphs and tables act as a fundamental support for the presentation of a scientific article. The images (simple and sometimes even self-describing) represent a great resource for the reader, as well as for the author. Reading becomes faster and more explanatory, as well as less demanding. Zaumanis provides the link to a site (https://peerrecognized.com) specifically for the creation of an article, it contains all the necessary tools in this regard.

E: Explain the results. While writing the article it is essential to always keep in mind the message you want to communicate; it must be direct and “easy” to understand, despite the complexity of the research carried out. Thus we move on to the explanation of the objectives, the results and the methods used (all this will be refined each time also from the linguistic point of view). The aim will be to make every reader come to the same conclusion. The importance of the sources and the bibliography consulted for the work should not be underestimated, it is important that users can verify the veracity of the research to deem it reliable (and then cite it and exploit it in other fields of study). Each initially set goal must find its own answer within the conclusion. The importance of consistency.

A: Advertise. The research requires a solid and coherent structure for a quality publication. This means that the structural order of the article will be relevant to the impact it will have on readers. Abstract and title will be essential to attract the reader’s attention and encourage him to read the paper. This means that an easy-to-understand abstract will be more stimulating for users.

P: Prepare for submission. The quality of the paper will certainly be relevant for a publication. What is even more important, however, is the selection of the right newspaper or magazine in which to publish it. There are some important selection criteria in this regard; objectives and history of the magazine/newspaper; required content; access to the public; periodicity; ranking and reliability.

Part 2: Know the rules of the game

Eight rules to follow for publication with the aim of achieving success and a high level of academic resonance.

Publish a lot at the beginning of your career: it is important to create a certain frequency with regard to publications. If you do not make mistakes (sometimes fatal), the result will be optimal and the fame will grow more and more.

Publish high-impact papers. Co-author efficiently: it is difficult and extremely important, in the case of collaborations, to find valid and collaborative co-authors at the right point. Knowledge of the other is the basis of everything. This is followed by the elaboration and definition of the objective. The division of tasks: always summarize all the decisions made by making written outlines.

Build an online presence to keep up with the times and with the technological resources that fortunately we have available. Creating a name will not be easy but a good online presentation of your person is an excellent springboard for the publication and dissemination of your research works.

Prioritize journals over conference proceedings: it is good that a research work is completed before any presentation of the same, during a conference. The presentation of the results obtained then, during debates and conferences, will make it possible to reach an even wider audience than that of the journals.

Advertise in conferences importance of peer recognition. The 5 S pyramid for presentation. Substance (at the base of any scientific presentation); Structure (if a thesis must respect the intro-method-results-conclusions structure; in a conference the important thing is to be clear and direct with the audience); Show Stories (short, evaluable and interesting to get your message across to the public); Speaker (presence and attitude are very important.

Publish open access increases credibility, allows possible collaborations; greater visibility from large companies and publishers; sharing of knowledge; more quotes.

Review others’ work. Be critical but always impartial! Criticism must be constructive, and must be able to produce more knowledge.

In conclusion, while reiterating that before deciding on which journal, even of a multidisciplinary type, it is better to publish, it is convenient to carefully consider:

• Editorial board.
• Reputation of the journal in its disciplinary field.
• Circulation of the magazine.
• Impact factor or belonging to class A.

Here are some useful databases with free or reserved access (to some universities) for information:

• Journal of Citation Reports (restricted access database) which evaluates scientific journals by processing statistics based on the analysis of citations.
• Scopus Journal Metrics (restricted access database) which provides a quick and transparent analysis of the progress of a journal.
• Scimago Journal Rank free database that generates statistics on article citations also at country level.
• Elsevier Journal Finder tool developed by Elsevier to select the most suitable journal starting from the abstract that is produced.
• JournalGuide free site useful for easy and reliable recognition of authoritative journals where from the title of a journal we can find links to: Journal website, Author instructions and Submission page.
• ThinkCheckSubmit site born from the initiative of some publishers to facilitate the researcher in choosing reliable journals.