| Both sides previous revision
Previous revision
|
Next revision
Both sides next revision
|
github_auth48_experiment [2019/02/22 10:05]
rsewikiadmin |
github_auth48_experiment [2021/02/05 11:32]
arusso |
| ===== Experiment: Using GitHub for AUTH48 ===== | ===== Experiment: Using GitHub for AUTH48 ===== |
| |
| ==== Draft Process ==== | This experiment has been run twice (during AUTH48 for RFC 8446 and RFC 8829). The idea was to use GitHub instead of email for AUTH48 state. Details below. |
| | |
| | ==== Draft Process in 2018 ==== |
| |
| - The author will create a repository with just the XML. We ask that the author not use their existing repository, for various reasons. | - The author will create a repository with just the XML. We ask that the author not use their existing repository, for various reasons. |
| * 2018-03-01: approved for publication as an RFC. (in MISSREF because of normative references that are not yet approved) | * 2018-03-01: approved for publication as an RFC. (in MISSREF because of normative references that are not yet approved) |
| |
| ==== Evaluation Criteria ==== | ==== Evaluation Criteria (defined in 2018) ==== |
| |
| === RPC Criteria === | === RPC Criteria === |
| |
| |
| === Feedback from Eric Rescorla (author, TLS) === | === Feedback from Eric Rescorla (author, RFC 8446) === |
| | <code> |
| We experimented with using Github for the publication of TLS 1.3. | We experimented with using Github for the publication of TLS 1.3. |
| The overall process was that Github was used primarily for feedback: | The overall process was that Github was used primarily for feedback: |
| |
| -Ekr | -Ekr |
| | </code> |
| |
| === Feedback from RFC Editor === | === Feedback from RFC Editor (in 2018) === |
| This is the consolidated feedback from the RFC Editor regarding the experiment in using GitHub to process the TLS 1.3 document. Please note that this feedback focuses entirely on the GitHub experience; all comments regarding editorial changes are off topic. Also, the markdown conversation (i.e., editing in markdown, choosing a particular flavor of markdown) is also outside the scope of the experiment and this feedback. | This is the consolidated feedback from the RFC Editor regarding the experiment in using GitHub to process the TLS 1.3 document. Please note that this feedback focuses entirely on the GitHub experience; all comments regarding editorial changes are off topic. Also, the markdown conversation (i.e., editing in markdown, choosing a particular flavor of markdown) is also outside the scope of the experiment and this feedback. |
| |
| |
| What Went Well: | What Went Well: |
| * The author seemed happier with the interactions with the RFC Editor. | * The author seemed happier with the interactions with the RFC Editor. |
| | |
| * It was easy for the RFC Editor to download a copy of the revised XML files. | |
| |
| | * It was easy for the RFC Editor to download a copy of the revised XML files. |
| |
| What Was Challenging: | What Was Challenging: |
| * This was not a good choice of documents to do an experiment with; the length, criticality, and number of early questions that had to be held until AUTH48 made the overall experience more complicated than it would have been otherwise (from the RFC Editor's perspective). In particular, typically a set of questions would be sent during AUTH state (earlier in the process); for this document, all questions were held for AUTH48 state, and this added to the perceived excessive number of initial changes. | * This was not a good choice of documents to do an experiment with; the length, criticality, and number of early questions that had to be held until AUTH48 made the overall experience more complicated than it would have been otherwise (from the RFC Editor's perspective). In particular, typically a set of questions would be sent during AUTH state (earlier in the process); for this document, all questions were held for AUTH48 state, and this added to the perceived excessive number of initial changes. |
| | |
| | * While the discrete list in GitHub of each change seemed to make the review easier for the author, it proved to be significantly more difficult for the RPC editor. Rather than have the changes all in one place, this required a lot of clicking to each individual change to review the comments. It was very time consuming. |
| |
| * While the discrete list in GitHub of each change seemed to make the review easier for the author, it proved to be significantly more difficult for the RPC editor. Rather than have the changes all in one place, this required a lot of clicking to each individual change to review the comments. It was very time consuming. | * Being on the "Watch" list resulted in a great deal of noise that the RPC editor had to work through. It was difficult to determine if comments and proposed changes were being directed to the RFC Editor or to the author. |
| |
| * Being on the "Watch" list resulted in a great deal of noise that the RPC editor had to work through. It was difficult to determine if comments and proposed changes were being directed to the RFC Editor or to the author. | * The RPC editor's AQs (Author Query) are often truncated in GitHub, which introduced some confusion on both the part of the author and the RPC editor. |
| |
| * The RPC editor's AQs (Author Query) are often truncated in GitHub, which introduced some confusion on both the part of the author and the RPC editor. | * In a document that has a large number of AQs (as this one did), the GitHub UI will hide the majority of those questions, along with some of the comments from authors and other reviewers (e.g., "120 hidden conversations"); one has to manually "unhide" them, 20 at a time. This process needs to be repeated every time you leave/return to that page. |
| |
| * In a document that has a large number of AQs (as this one did), the GitHub UI will hide the majority of those questions, along with some of the comments from authors and other reviewers (e.g., "120 hidden conversations"); one has to manually "unhide" them, 20 at a time. This process needs to be repeated every time you leave/return to that page. | * There was definitely confusion about how the RPC editors actually edit a document reflected in a request to receive intermediate diff files that show one type of change (header changes only, whitespace changes only, editorial changes only, etc). That is not how documents are edited, and trying to do that would definitely increase the level or work required on the part of the RPC editor. |
| |
| * There was definitely confusion about how the RPC editors actually edit a document reflected in a request to receive intermediate diff files that show one type of change (header changes only, whitespace changes only, editorial changes only, etc). That is not how documents are edited, and trying to do that would definitely increase the level or work required on the part of the RPC editor. | * "Reflowed" text within the XML file is a common result of making editorial changes or inserting questions into the XML file. Typically, this has not been a concern for authors or the RPC, as such changes aren't reflected in the publication output, and the only diffs reviewed are text file vs. text file. In the case of this document, when reviewing XML diffs generated by GitHub, this introduced a number of changes that were noise. In the future, one way to avoid this would be careful editing of the source file to avoid reflowing the text or inserting comments mid-paragraph; however that seems like a lot of work for little return given the purpose behind the XML file. (As an aside to this one, in the future, improved XML diffing might not display changes to reflowed text in the XML -- the xmldiff tool is under development as part of the format work.) |
| |
| * "Reflowed" text within the XML file is a common result of making editorial changes or inserting questions into the XML file. Typically, this has not been a concern for authors or the RPC, as such changes aren't reflected in the publication output, and the only diffs reviewed are text file vs. text file. In the case of this document, when reviewing XML diffs generated by GitHub, this introduced a number of changes that were noise. In the future, one way to avoid this would be careful editing of the source file to avoid reflowing the text or inserting comments mid-paragraph; however that seems like a lot of work for little return given the purpose behind the XML file. (As an aside to this one, in the future, improved XML diffing might not display changes to reflowed text in the XML -- the xmldiff tool is under development as part of the format work.) | |
| |
| | Proposed Changes to the next stage of the GitHub Experiment (JSEP draft)\\ |
| | [Full detail of the process is on this page] |
| |
| Proposed Changes to the next stage of the GitHub Experiment (JSEP draft) | * change to step 5 - rather than having the RFC Editor subscribe to all notifications, have the author(s) add an @mention when the RFC Editor needs to do something. (Assuming this feature will work if the account is not actually watching the repository.) |
| Full detail of the process is here: https://www.rfc-editor.org/rse/wiki/doku.php?id=github_auth48_experiment | |
| * change to step 5 - rather than having the RFC Editor subscribe to all notifications, have the author(s) add an @mention when the RFC Editor needs to do something. (Assuming this feature will work if the account is not actually watching the repository.) | |
| |
| * change to step 10- rather than the RFC Editor emailing the link to the latest XML file to the author, the RPC editor will submit it directly to GitHub. | * change to step 10- rather than the RFC Editor emailing the link to the latest XML file to the author, the RPC editor will submit it directly to GitHub. |
| |
| |
