POPL Research Papers

Accepted Papers

The proceedings of POPL 2024 are freely available as a special issue (Volume 8, No. POPL) of the PACMPL journal.

POPL 2024 Call for Papers

PACMPL Issue POPL 2024 seeks contributions on all aspects of programming languages and programming systems, both theoretical and practical. Authors of papers published in PACMPL Issue POPL 2024 will be invited to present their work in the POPL conference in January 2024, which is sponsored by ACM SIGPLAN, in cooperation with ACM SIGACT and ACM SIGLOG.

Scope

Principles of Programming Languages (POPL) is a forum for the discussion of all aspects of programming languages and programming systems. Both theoretical and experimental papers are welcome. We seek submissions that make principled, enduring contributions to the theory, design, understanding, implementation, or application of programming languages.

Evaluation Criteria

The Program Committee will evaluate the technical contribution of each submission as well as its accessibility to both experts and the general POPL audience. All papers will be judged on significance, originality, relevance, correctness, and clarity. Each paper must explain its scientific contribution in both general and technical terms, identifying what has been accomplished, explaining why it is significant, and comparing it with previous work. Advice on writing technical papers can be found on the SIGPLAN author information page.

Deadlines and formatting requirements, detailed below, will be strictly enforced.

Full Double-Blind Reviewing Process

POPL 2024 will use a full double-blind reviewing process (similar to the one used for POPL 2023 but different from the lightweight double-blind process used in previous years). This means that identities of authors will not be made visible to reviewers until after conditional-acceptance decisions have been made, and then only for the conditionally-accepted papers. The use of full double-blind reviewing has several consequences for authors.

• Submissions: Authors must omit their names and institutions from their paper submissions. In addition, references to authors’ own prior work should be in the third person (e.g., not “We build on our previous work …” but rather “We build on the work of …”).

• Supplementary material: Authors are permitted to provide supplementary material (e.g., detailed proofs, proof scripts, system implementations, or experimental data) along with their submission, which reviewers may (but are not required to) examine. This material may take the form of a single file, such as a PDF or a tarball. Authors must fully anonymize any supplementary material. Links to supplementary material on external websites are not permitted.

• Author response: In responding to reviews, authors should not say anything that reveals their identity, since author identities will not be revealed to reviewers at that stage of the reviewing process.

• Dissemination of work under submission: Authors are welcome to disseminate their ideas and post draft versions of their paper(s) on their personal website, institutional repository, or arXiv (reviewers will be asked to turn off arXiv notifications during the review period). But authors should not take steps that would almost certainly reveal their identities to members of the Program Committee, e.g., directly contacting PC members or publicizing the work on widely-visible social media or major mailing lists used by the community.

The purpose of the above restrictions is to help the Program Committee and external reviewers come to a judgment about the paper without bias, not to make it impossible for them to discover the authors’ identities if they were to try. In particular, nothing should be done in the name of anonymity that weakens the quality of the submission.

However, there are occasionally cases where adhering to the above restrictions is truly difficult or impossible for one reason or another. In such cases, the authors should contact the Program Chair to discuss the situation and how to handle it.

The FAQ on Double-Blind Reviewing addresses many common scenarios and answers many common questions about this topic. But there remain many grey areas and trade-offs. If you have any doubts about how to interpret the double-blind rules or you encounter a complex case that is not clearly covered by the FAQ, please contact the Program Chair for guidance.

Additional Details of the Reviewing Process

POPL 2024 will have four Associate Chairs who will help the Program Chair monitor reviews, solicit external expert reviews for submissions when there is not enough expertise on the committee, and facilitate reviewer discussions.

As in previous years, authors will have a multi-day period to respond to reviews, as indicated in the Important Dates table. Responses are optional but recommended. A response should address specific points or questions raised in the reviews as well as any erroneous claims made by reviewers; in particular, it should not present new technical results, unless specifically requested by the reviewers.

The Program Committee (PC) will discuss papers electronically using the HotCRP conference management system. There will be no physical or synchronous PC meeting; this will avoid the time, cost, and environmental impact of transporting an increasingly large committee to one point on the globe. There is also no formal External Program Committee or Extended Review Committee, though experts outside the committee will be consulted as needed. Paper decisions will be accompanied by a “meta-review” summarizing the reasoning behind the committee’s decision.

To conform with ACM requirements for journal publication, all POPL papers will be subject to two rounds of review. At the end of the first round, the PC will select a set of conditionally accepted papers, each with a clear list of mandatory revisions that the authors must implement in order for the paper to be accepted. For the second round, authors of conditionally accepted papers will then be required to submit a short description of how they have revised the paper, including how the mandatory revisions have been implemented. The PC will then check the revisions and make final acceptance decisions. Authors of conditionally accepted papers must submit a satisfactory revision to the PC by the second-round deadline or risk rejection.

For additional information about the reviewing process, see: Principles of POPL, a presentation of the underlying organizational and reviewing policies for POPL. For POPL 2024, policies specified in this Call for Papers supersede those in the Principles of POPL document.

Submission Site Information

The submission site is https://popl24.hotcrp.com.

Authors can submit multiple times prior to the deadline. Only the last submission will be reviewed. There is no abstract deadline. The submission site requires entering author names and affiliations, relevant topics, and potential conflicts. Addition or removal of authors after the submission deadline will need to be approved by the Program Chair (as this kind of change potentially undermines the goal of eliminating conflicts during paper assignment).

The submission deadline is 11:59 PM on July 11, 2023, Anywhere on Earth: https://en.wikipedia.org/wiki/Anywhere_on_Earth

Conflicts of Interest

For each submission, the authors must make sure that they properly declare all potential conflicts of interest for all of the authors of that submission. This includes marking PC conflicts as well as “Other Conflicts (external)”. A conflict caught late in the reviewing process leads to a voided review which may be infeasible to replace.

Conflicts should be declared between an adviser and an advisee (e.g., Ph.D., post-doc; forever), between an author and a co-author (papers and proposals; for two years), between people at the same institution (branches of large companies or different locations of research institutes are considered to be the same institution; for two years after leaving an institution), between people with financial conflicts of interest, and between relatives.

Please do not declare spurious conflicts: such incorrect conflicts are especially harmful if the aim is to exclude potential reviewers, so spurious conflicts can be grounds for rejection. If you are unsure about a conflict, please ask the Program Chair.

Submission Guidelines

Prior to the paper submission deadline, authors should upload their full anonymized paper. Here are some key requirements concerning paper submissions:

• Each paper should have no more than 25 pages of text, excluding bibliography, using the PACMPL format (specifically, the acmart LaTeX class with acmsmall option). It is a single-column page layout with a 10 pt font, 12 pt line spacing, and wider margins than recent POPL page layouts. In this format, the main text block is 5.478 in (13.91 cm) wide and 7.884 in (20.03 cm) tall. Use of a different format (e.g., smaller fonts or a larger text block) is grounds for summary rejection. The PACMPL template for LaTeX can be found at the SIGPLAN author information page, and further information about PACMPL submissions can be found on the PACMPL author guidelines page. PACMPL does not support submissions in Microsoft Word.

• We strongly encourage use of the review and screen options in order to make submissions easier to review.

• We strongly encourage use of author-year citations, since that is the citation format required by PACMPL for final versions of accepted papers.

• Submissions should be in PDF and printable on both US Letter and A4 paper. Papers may be resubmitted to the submission site multiple times up until the deadline, but the last version submitted before the deadline will be the version reviewed.

• Submitted papers must adhere to the SIGPLAN Republication Policy and the ACM Policy on Plagiarism. Concurrent paper submissions to other conferences, workshops, journals, or similar forums of publication are not allowed.

• Authors are free to submit supplementary material along with their submissions, but it must be fully anonymized.

• Authors must list all their conflicts of interest (both PC conflicts and external conflicts) in the HotCRP submission form.

• Authors may include additional information in a field of the HotCRP submission form labeled “Confidential Comments for the Program Chair”. This information need not be anonymized. It can be used to inform the Program Chair, for example, about sensitive issues concerning a conflict with a PC member or about supplementary material that cannot be anonymized. It is left to the discretion of the Program Chair what to do with this information.

• If for some reason an author feels uncomfortable discussing a sensitive issue with the Program Chair (or communicating via the “Confidential Comments” field in HotCRP), they should feel free to get in touch instead with any of the Associate Chairs, with whom they can discuss the issue in confidence.

• Submissions from PC members (except the Program Chair) are permitted and will not be handled any differently than other submissions. This is in accordance with a recent change in policy approved by the SIGPLAN Executive Committee: SIGPLAN conferences that use full double-blind review and whose PCs have at least 50 members need not hold PC submissions to a higher standard.

Artifact Evaluation for Accepted Papers

Authors of conditionally accepted papers will be invited to formally submit supporting materials to the Artifact Evaluation process. Artifact Evaluation is run by a separate committee whose task is to assess how the artifacts support the work described in the papers. Artifact submission is strongly encouraged but voluntary and will not influence the final decision regarding the papers. Papers that go through the Artifact Evaluation process successfully will receive a seal of approval printed on the papers themselves. Authors of accepted papers are encouraged to make these materials publicly available upon publication of the proceedings, by including them as “source materials” in the ACM Digital Library.

Copyright, Publication, and Presentation

As a Gold Open Access journal, PACMPL is committed to making peer-reviewed scientific research free of restrictions on both access and (re-)use. Authors are strongly encouraged to support libre open access by licensing their work with the Creative Commons Attribution 4.0 International (CC BY) license, which grants readers liberal (re-)use rights.

Authors of accepted papers will be required to choose one of the following publication rights:

• Author licenses the work with a Creative Commons license, retains copyright, and (implicitly) grants ACM non-exclusive permission to publish (suggested choice).
• Author retains copyright of the work and grants ACM a non-exclusive permission to publish license.
• Author retains copyright of the work and grants ACM an exclusive permission to publish license.
• Author transfers copyright of the work to ACM.

These choices follow from ACM Copyright Policy and ACM Author Rights, corresponding to ACM’s “author pays” option. While PACMPL may ask authors who have funding for open-access fees to voluntarily cover the article processing charge (currently, US$400), payment is not required for publication. PACMPL and SIGPLAN continue to explore the best models for funding open access, focusing on approaches that are sustainable in the long-term while reducing short-term risk.

All papers will be archived by the ACM Digital Library. Authors will have the option of including supplementary material with their paper. The official publication date is the date the proceedings are made available in the ACM Digital Library. This date may be up to two weeks prior to the first day of the conference. The official publication date affects the deadline for any patent filings related to published work.

Authors of accepted papers will be given the opportunity to present their work at the conference in the form of a short talk. It is recommended that they give a talk in person at the conference, so that everyone can get the most out of the conference experience. However, authors who cannot attend in person will be provided with some option for remote presentation, as well as some mechanism for remote interaction with conference participants. All presenting authors will be required to register for the conference. Authors also have the option of not presenting their work at the conference, in which case they do not need to register.

Distinguished Paper Awards

At most 10% of the accepted papers of POPL 2024 will be designated as Distinguished Papers. This award highlights papers that the Program Committee thinks should be read by a broad audience due to their relevance, originality, significance, and clarity. The selection of the distinguished papers will be made based on the final version of the paper and through an additional review process.

General

Q: Why are you using double-blind reviewing?

A: Studies have shown that a reviewer’s attitude toward a submission may be affected, even unconsciously, by the identity of the authors. We want reviewers to be able to approach each submission without any such, possibly involuntary, pre-judgment. Many computer science conferences have embraced double-blind reviewing. POPL has used double-blind reviewing for several years now as stipulated in the Principles of POPL.

Q: Why are you using full double-blind reviewing instead of lightweight double-blind reviewing?

A: For about a decade, POPL used lightweight double-blind review (double-blind until PC members submit reviews). For POPL 2023, the conference switched to full double-blind review (double-blind until conditional acceptance). There are benefits and drawbacks to both approaches, but based on our experience so far, the benefits of full-double blind seem to outweight the drawbacks.

Q: Do you really think blinding actually works? I suspect reviewers can often guess who the authors are anyway.

A: It is relatively rare for authorship to be guessed correctly, even by expert reviewers, as detailed in this study.

Q: Couldn’t blind submission create an injustice where a paper is inappropriately rejected based upon supposedly-prior work which was actually by the same authors and not previously published?

A: Yes, but we have mechanisms in place to prevent such an injustice from occurring. Reviewers are held accountable for their positions and are required to identify any supposed prior work that they believe undermines the novelty of the paper. Any assertion that “this has been done before” by reviewers should be supported with concrete information. The author response mechanism exists in part to hold reviewers accountable for claims that may be incorrect. Moreover, if authors believe that their paper is being misjudged due to the constraints of double-blind review, they should feel free to get in touch with the Program Chair.

For authors

Q: What exactly do I have to do to anonymize my paper?

A: Use common sense. Your job is not to make your identity undiscoverable but simply to make it possible for reviewers to evaluate your submission without having to know who you are. The specific guidelines stated in the call for papers are simple: omit authors’ names from your title page, and when you cite your own work, refer to it in the third person. For example, if your name is Smith and you have worked on amphibious type systems, instead of saying “We extend our earlier work on statically typed toads [Smith 2004],” you might say “We extend Smith’s [2004] earlier work on statically typed toads.” Also, be sure not to include any acknowledgements that would give away your identity, and avoid revealing the institutional affiliation of authors or at which the work was performed.

If you believe for some reason that it is impossible to anonymize your paper without significantly weakening it, please feel free to get in touch with the Program Chair to discuss the situation.

Q: I would like to provide supplementary material for consideration, e.g., the code of my implementation or proofs of theorems. How do I do this?

A (and also see the next question): On the submission site there will be an option to submit supplementary material along with your main paper. This supplementary material should also be anonymized; it may be viewed by reviewers during the review period, so it should adhere to the same double-blind guidelines. Please do not provide supplementary materials via links to external web sites. It is possible to change the linked items after the submission deadline has passed, and, to be fair to all authors, we would like to be sure reviewers evaluate materials that have been completed prior to the submission deadline.

Q: My submission is based on code available in a public repository or I would like to provide a link to an online demo. How do I deal with this?

A: Authors should not submit links to public repositories as supplementary material. In general, most code in a public repository can be repackaged and anonymized and submitted as a tarball at the time of paper submission. (Note: it is only necessary to anonymize new code that is part of the contribution of the paper, not pre-existing code or libraries on which one is building.)

However, there may be cases where this anonymization is for some reason difficult or impossible. In that case, authors should explain this in the “Confidential Comments for the Program Chair” field of their submission and include the link to a public repository there. The Program Chair and Associate Chairs will then decide (on a case-by-case basis) whether it is appropriate to share this link with reviewers. Needless to say, attempting to discover the reviewers for your paper by tracking visitors to a public repository would constitute a breach of academic integrity.

The same goes for online demos. A link to an online demo should only be given in the Confidential Comments for the Program Chair.

Q: I am building on my own past work on the WizWoz system. Do I need to rename this system in my paper for purposes of anonymity, so as to remove the implied connection between my authorship of past work on this system and my present submission?

A: No. The relationship between systems and authors changes over time, so there will be at least some doubt about authorship. Increasing this doubt by changing the system name would help with anonymity, but it would compromise the research process. In particular, changing the name requires explaining a lot about the system again because you can’t just refer to the existing papers, which use the proper name. Not citing these papers runs the risk of the reviewers who know about the existing system thinking you are replicating earlier work. It is also confusing for the reviewers to read about the paper under Name X and then have the name be changed to Name Y. Will all the reviewers go and re-read the final version with the correct name? If not, they have the wrong name in their heads, which could be harmful in the long run.

Q: I am submitting a paper that extends my own work that previously appeared at a workshop. Should I anonymize any reference to that prior work?

A: If the previous workshop publication resulted in a published paper, then your POPL submission will be judged on the significance of its delta with respect to that earlier paper. In that case, you should definitely cite the earlier paper as you would any other paper (i.e., in the third person, not revealing that you were the author of the earlier paper) and explain the delta. If, on the other hand, the work presented at an earlier workshop was not accompanied by a proper publication (e.g., if it was just a talk), then your POPL submission may very well have significant overlap with it and citing the workshop work would effectively unblind your paper. In such a case, it is not necessary to cite the earlier presentation. Instead, please mention it in the “Confidential Comments for the Program Chair” field of the HotCRP form so that the Program Chair is aware of the situation. When in doubt, ask the Program Chair for guidance.

Q: I want to cite some related work that itself cites an earlier version of my paper/system that appeared previously online. But that would effectively unblind my submission. What do I do?

A: It is difficult to give a general answer to this question. Ask the Program Chair for guidance.

Q: My submission presents a technique that I employed in the development of previous papers but which was never properly described in those papers. If I mention that my technique was used in those previous papers, I am effectively unblinding my submission, but if I don’t mention it, reviewers may think I am ripping off my own prior work. What do I do?

A: It is difficult to give a general answer to this question. Ask the Program Chair for guidance.

Q: Am I allowed to post my (non-blinded) paper on my web page? Can I advertise the unblinded version of my paper on mailing lists or send it to colleagues? Can I give a talk about my work while it is under review? How do I handle social media? What about arXiv?

A: We have developed guidelines, described here, to help everyone navigate in the same way the tension between (1) the normal communication of scientific results, which double-blind reviewing should not impede, and (2) actions that essentially force potential reviewers to learn the identity of the authors for a submission. Roughly speaking, you may (of course!) discuss work under submission, but you should not broadly advertise your work through media that is likely to reach your reviewers. We acknowledge there are grey areas and trade-offs; we cannot describe every possible scenario.

Things you may do:

• Submit your paper to POPL, even if a previous version of it – under the same title or a different title – has been presented at an informal workshop, published on arXiv, or submitted to a previous conference or workshop.
• Post your submission on your home page, your institutional repository, or arXiv, before or after the deadline (under whatever title you want).
• Discuss your work with anyone who is not on the PC, or with PC members with whom you already have a conflict.
• Present your work at professional meetings, informal workshops, or job interviews during the POPL review period.

The above is not an exhaustive list: when in doubt, ask the Program Chair.

Things you should not do:

• Contact PC members (with whom you are not conflicted) about your work.
• Publicize your work on major mailing lists used by the community (because potential reviewers likely read these lists).
• Publicize your work on social media if wide public [re-]propagation is common (e.g., Twitter) and therefore likely to reach potential reviewers. For example, on Facebook, a post with a broad privacy setting (public or all friends) saying, “Whew, POPL paper in, time to sleep” is okay, but one describing the work or giving its title is not appropriate. Alternately, a post to a group including only the colleagues at your institution is fine.

Reviewers will not be asked to recuse themselves from reviewing your paper unless they feel you have gone out of your way to advertise your authorship information to them. If you are unsure about what constitutes “going out of your way”, please contact the Program Chair.

Q: Will the fact that POPL is double-blind have an impact on handling conflicts-of-interest?

A: Double-blind reviewing does not change the principle that reviewers should not review papers with which they have a conflict of interest, even if they do not immediately know who the authors are. Authors declare conflicts of interest when submitting their papers using the guidelines in the call for papers. Papers will not be assigned to reviewers who have a conflict.

For reviewers

Q: What should I do if I learn the authors’ identity? What should I do if a prospective POPL author contacts me and asks to visit my institution?

A: You should not treat double-blind reviewing differently from other reviewing. In particular, as explained above, it is fine for authors to give talks about their work (at workshops, job interviews, etc.), they cannot control who will attend their talks, and you as a PC member should not feel that you are prevented from attending their talks. Knowing (for whatever reason) that a paper was written by a certain author does not prevent you from reviewing the paper.

That said, if you feel that the authors’ actions are in flagrant violation of double-blind review, that is a point of concern. For example, if an author e-mails you their brand new POPL submission and asks to visit your institution to discuss it, that would clearly not be appropriate. There is a grey zone here, so use your best judgment, and when in doubt, ask the Program Chair how to proceed.

Q: The authors have provided a URL to supplemental material. I would like to see the material but I worry they will snoop my IP address and learn my identity. What should I do?

A: As explained above, authors should not do this. If you discover that they have done so, please bring it to the attention of the Program Chair.

Q: If I am assigned a paper for which I feel I am not an expert, how do I seek an outside review?

A: PC members should do their own reviews, not delegate them to someone else. If doing so is problematic for some papers, e.g., you don’t feel completely qualified to review them, or if you know someone who would be the perfect reviewer for the paper, then bring this to the attention of the Associate Chair handling your paper, and discuss with them potential external reviewers (or other PC members) who might be able to provide an expert review. Then, submit a review for the paper that is as careful as possible, pointing out the areas where you think your knowledge is lacking. The Associate Chairs will manage the process of ensuring that papers receive sufficient expert reviews; please do not contact outside reviewers yourself.

Q: How do we handle potential conflicts of interest since I cannot see the author names?

A: The conference review system will ask that you identify conflicts of interest when you get an account on the submission system. It is critical that you enter these at least a week before the POPL submission deadline. Feel free to also identify additional authors whose papers you feel you could not review fairly for reasons other than those given (e.g., strong personal friendship).

Q: How should I avoid learning the authors’ identity if I am using web-search in the process of performing my review?

A: You should make a good-faith effort not to find the authors’ identity during the review period, but if you inadvertently do so, this does not disqualify you from reviewing the paper. As part of the good-faith effort, avoid using search engines with terms like the paper’s title or the name of a new system being discussed.

These guidelines are largely based on guidelines for POPL 2023, with a few modifications by Derek Dreyer based on discussions with the POPL Steering Committee. The earlier guidelines evolved from guidelines used at POPL and PLDI for a number of years, which originated in guidelines created by Mike Hicks for POPL 2012.