This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
github_auth48_experiment [2019/02/22 10:03] rsewikiadmin added feedback from author, RFC Ed |
github_auth48_experiment [2021/02/24 00:15] (current) arusso |
||
---|---|---|---|
Line 1: | Line 1: | ||
===== Experiment: Using GitHub for AUTH48 ===== | ===== Experiment: Using GitHub for AUTH48 ===== | ||
- | ==== Draft Process ==== | + | This experiment has been run twice (during AUTH48 for RFC 8446 and RFC 8829). The idea was to use GitHub instead of email for AUTH48 state. Details below. |
+ | |||
+ | ^ ^ RFC 8446 | ||
+ | | I-D | ||
+ | | Pages submitted | 156 pages | 115 pages | | ||
+ | | I-D approved | ||
+ | | AUTH48 start | ||
+ | | Publication | ||
+ | | GitHub repo | https:// | ||
+ | ^ AUTH48 details | ||
+ | | Time in state | 8.2 weeks | 28.3 weeks | | ||
+ | | # questions at start | 58 | 60\\ (yielded [[https:// | ||
+ | |||
+ | ==== Process | ||
- The author will create a repository with just the XML. We ask that the author not use their existing repository, for various reasons. | - The author will create a repository with just the XML. We ask that the author not use their existing repository, for various reasons. | ||
Line 9: | Line 22: | ||
* status of this memo | * status of this memo | ||
* section movement to meet the style guide requirements (e.g., move acks and contribs to appear at the end) | * section movement to meet the style guide requirements (e.g., move acks and contribs to appear at the end) | ||
- | - RFC Editor goes through the EDIT and RFC EDITOR process states and sends an email with a link to the XML file created in step 2 and the edited XML file at the start of AUTH48. | + | - RFC Editor goes through the EDIT and RFC EDITOR process states and sends an email with a link to the XML file created in step 2 and the edited XML file at the start of AUTH48. |
- At the start of AUTH48, the author will create a Pull Request (PR) with the changes proposed by the RPC. The authors respond and point to the repository they created. | - At the start of AUTH48, the author will create a Pull Request (PR) with the changes proposed by the RPC. The authors respond and point to the repository they created. | ||
- The relevant people from the RPC will subscribe to notifications from that repository using the “Watch” button and confirm they are watching. For this test case, Heather, Adam, and Martin will follow along as well. | - The relevant people from the RPC will subscribe to notifications from that repository using the “Watch” button and confirm they are watching. For this test case, Heather, Adam, and Martin will follow along as well. | ||
Line 21: | Line 34: | ||
- Final approval for the full document needs to result in an explicit email to the the RPC from each author indicating approval. GitHub will be used for editorial discussion, but the final text approval will be done via email. | - Final approval for the full document needs to result in an explicit email to the the RPC from each author indicating approval. GitHub will be used for editorial discussion, but the final text approval will be done via email. | ||
- | ==== Relevant Documents ==== | + | ==== Evaluation Criteria |
- | + | ||
- | * **[[https:// | + | |
- | * 2018-03-21: approved for publication as an RFC. | + | |
- | * 2018-06-14: initiated AUTH48. provided XML file as described | + | |
- | * 2018-08-10: published as RFC 8446 | + | |
- | * **[[https:// | + | |
- | * 2018-03-01: approved for publication as an RFC. (in MISSREF because of normative references that are not yet approved) | + | |
- | + | ||
- | ==== Evaluation Criteria | + | |
=== RPC Criteria === | === RPC Criteria === | ||
- | Does GitHub seem to provide | + | Does GitHub seem to provide |
* tracking changes during AUTH48 in such a way that the RPC can query, at any point of time in the future, who approved those changes? | * tracking changes during AUTH48 in such a way that the RPC can query, at any point of time in the future, who approved those changes? | ||
* having clear interactions with all parties that need to submit approvals during AUTH48? | * having clear interactions with all parties that need to submit approvals during AUTH48? | ||
Line 40: | Line 44: | ||
=== Measurements === | === Measurements === | ||
- | * Average time in AUTH48 state for standard process documents: | + | * Average time in AUTH48 state for standard process documents |
- | * Average time in AUTH48 state for 100+ page standard process documents: | + | * Average time in AUTH48 state for 100+ page standard process documents |
- | * Time in AUTH48 state for this document: | + | * Time in AUTH48 state for this document |
- | * Number of questions at start of AUTH48: | + | * Number of questions at start of AUTH48 |
- | * Number of pull requests afterwards: | + | * Number of pull requests afterwards |
- | For RFC 8446: | + | === Measurements in 2018 and 2020 === |
- | * Average | + | ^ ^ RFC 8446 ^ RFC 8829 ^ |
- | * average times in AUTH48 | + | | AUTH48 |
- | | + | ^ Comparing with standard process |
- | * May 1 report: 3.2 weeks | + | | Concurrent Avg. AUTH48 |
- | * Apr 1 report: 3.4 weeks | + | | Avg. AUTH48 |
- | * Average time in AUTH48 state for 100+ page standard process documents: | + | |
- | * Average 7.7 weeks in AUTH48 | + | |
- | | + | |
- | * Note: half of the 100+ page documents had an AUTH state (and for those that did, 3.2 weeks were in that state) | + | |
- | * Time in AUTH48 state for this document: | + | |
- | * 8.2 weeks | + | |
- | * Number of questions at start of AUTH48: 58 | + | |
- | [[https:// | + | |
- | === Notes === | ||
- | * future work will require mapping of GitHub account names to Authors, WG Chairs, Document Shepherds, ADs, Stream Managers (possible GDPR implications this information is outside the publishing industry norm for information stored about an individual during a publication process) | ||
+ | * using standard AUTH48 process for the 3 months preceding publication.\\ | ||
+ | < | ||
- | === Feedback from Eric Rescorla (author, TLS) === | ||
- | We experimented with using Github for the publication of TLS 1.3. | ||
- | The overall process was that Github was used primarily for feedback: | ||
- | 1. The RFC Editor supplied iterative versions of the XML files. | + | === Notes === |
- | 2. The draft editor put them up on Github (initially as PRs | + | |
- | and later just as commits) | + | |
- | 3. We used Github' | + | |
- | from the RPC and from the editor/ | + | |
- | 4. The editor provided updated XML from the editor/ | + | |
- | in three ways: | + | |
- | (a) patches | + | |
- | (b) new XML files | + | |
- | (c) OLD/NEW format | + | |
- | + | ||
- | + | ||
- | WHAT WENT WELL | + | |
- | I found Github very useful for refining the precise content of what | + | |
- | we plan to ship. Specifically: | + | |
- | + | ||
- | - The RPC makes a lot of changes in their first pass, and so | + | |
- | | + | |
- | each one. | + | |
- | + | ||
- | - As the editor/ | + | |
- | that interaction and let us only ship when we had refined text. | + | |
- | + | ||
- | Having each XML revision in Github also made it possible to do | + | |
- | incremental diffs, which is inconvenient with the current practice | + | |
- | of updating the XML and TXT files on the FTP site with every revision. | + | |
- | In one case, it let us track down when a particular error was | + | |
- | introduced, which was helpful. | + | |
- | + | ||
- | + | ||
- | WHAT WENT BADLY | + | |
- | The initial AUTH48 version from the RFC Editor included both textual | + | |
- | changes and whitespace changes. This produced a very large diff which | + | |
- | was difficult to work with. In future, it would work better to have as | + | |
- | many intermediate versions | + | |
- | stage of the process) so that they can be individually reviewed. | + | |
- | + | ||
- | Having the RPC email out new XML files and then the editor upload | + | |
- | them is clunky. It would work better if the RPC uploaded them directly | + | |
- | (probably as PRs, though this is TBD). We adopted this strategy | + | |
- | minimize overhead for the RPC, but it's easy to get lost. Conversely, | + | |
- | having the editor mail new XML to the RPC is clunky. This would | + | |
- | work better if Github were the master copy, which is how we do | + | |
- | things in WGs. | + | |
- | + | ||
- | + | ||
- | + | ||
- | THOUGHTS FOR FUTURE EXPERIMENTS | + | |
- | This was valuable for me. In previous RFCs I have done it has been | + | |
- | quite hard to keep track of all the changes and this was a bigger | + | |
- | project. As noted above, I found it very useful to be able to | + | |
- | individually see address each change that was made in AUTH48. | + | |
- | However, this hybrid approach where the editor is the interface | + | |
- | between the RPC and Github is awkward. | + | |
- | + | ||
- | If we try this again -- and I think we should -- I think we should do | + | |
- | one of two things: | + | |
- | + | ||
- | (a) Go full Github and use it the way WGs do. | + | |
- | (b) Limit the RPC's use of Github to seeing the editor comments on | + | |
- | their changes. | + | |
- | + | ||
- | From the document editor perspective, | + | |
- | seems like a big jump for the RPC, so we should probably look at (b). | + | |
- | I think the easiest way to do (b) would just be to have two repos: | + | |
- | + | ||
- | (a) the master repo which is used just to integrate the RPC's changes | + | |
- | (b) a second repo (the editor' | + | |
- | the changes. | + | |
- | + | ||
- | Then we could treat the RPC's changes as PRs against the repo (as | + | |
- | I had originally intended) but they wouldn' | + | |
- | noise from the editor/ | + | |
- | + | ||
- | Ben also observed that because so much of the value of the RPC-editor | + | |
- | Github interaction is the first set of changes from the RPC (because | + | |
- | that's where most of the comments need to be made) that we could | + | |
- | skip Github entirely for that and just do the comments in Phabricator. | + | |
- | That seems like yet another tool people have to learn, though, so | + | |
- | I tend to think we should stick with Github for that. | + | |
- | + | ||
- | + | ||
- | -Ekr | + | |
- | + | ||
- | === Feedback from RFC Editor === | + | |
- | This is the consolidated feedback from the RFC Editor regarding the experiment in using GitHub to process the TLS 1.3 document. Please note that this feedback focuses entirely on the GitHub experience; all comments regarding editorial changes are off topic. Also, the markdown conversation (i.e., editing in markdown, choosing a particular flavor of markdown) | + | |
- | + | ||
- | tl;dr | + | |
- | GitHub did not improve anything | + | |
- | + | ||
- | + | ||
- | What Went Well: | + | |
- | * The author seemed happier with the interactions with the RFC Editor. | + | |
- | + | ||
- | * It was easy for the RFC Editor to download a copy of the revised XML files. | + | |
- | + | ||
- | + | ||
- | What Was Challenging: | + | |
- | * This was not a good choice of documents to do an experiment with; the length, criticality, | + | |
- | + | ||
- | * While the discrete list in GitHub of each change seemed to make the review easier for the author, it proved to be significantly more difficult for the RPC editor. Rather than have the changes all in one place, this required a lot of clicking to each individual change to review the comments. It was very time consuming. | + | |
- | + | ||
- | * Being on the " | + | |
- | * The RPC editor' | + | ==== Feedback |
- | * In a document that has a large number of AQs (as this one did), the GitHub UI will hide the majority of those questions, along with some of the comments from authors and other reviewers (e.g., "120 hidden conversations" | + | |
- | * There was definitely confusion about how the RPC editors actually edit a document reflected in a request to receive intermediate diff files that show one type of change (header changes only, whitespace changes only, editorial changes only, etc). That is not how documents are edited, and trying to do that would definitely increase the level or work required on the part of the RPC editor. | + | -------------------------------- |
- | * " | + | ==== Work Flow in 2020-2021 ==== |
+ | The authors and editor agreed to use the GitHub Pull Request work flow: | ||
- | Proposed Changes to the next stage of the GitHub Experiment (JSEP draft) | + | - The authors answer |
- | Full detail | + | - The authors indicate that an issue is ready for editor attention by labeling the issue " |
- | * change | + | |
+ | - The editor makes edits in that branch. | ||
+ | - The editor commits changes. | ||
+ | - The editor creates a Pull Request | ||
+ | - The authors review | ||
+ | - If the PR is accepted, | ||
+ | - If the PR is rejected, the editor makes changes to the current branch and resubmits the PR. | ||
+ | - Repeat for the other issues. | ||
+ | |||
+ | The authors and the editor set up notifications so that many of these steps automatically generated emails to involved parties (assigning/ | ||
- | * change to step 10- rather than the RFC Editor emailing the link to the latest XML file to the author, the RPC editor will submit it directly to GitHub. | + | ==== Feedback in 2020-2021 ==== |
+ | * See [[github_exp_2021_feedback]] for notes from the RFC Editor. |