User Tools

Site Tools


github_auth48_experiment

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision
Previous revision
github_auth48_experiment [2018/06/13 17:53]
arusso created
github_auth48_experiment [2021/02/24 08:15] (current)
arusso
Line 1: Line 1:
 ===== Experiment: Using GitHub for AUTH48 ===== ===== Experiment: Using GitHub for AUTH48 =====
  
-=== Draft Process ===+This experiment has been run twice (during AUTH48 for RFC 8446 and RFC 8829). The idea was to use GitHub instead of email for AUTH48 state. Details below. 
 + 
 +^                ^   RFC 8446    RFC 8829  ^ 
 +| I-D          [[https://datatracker.ietf.org/doc/draft-ietf-tls-tls13|draft-ietf-tls-tls13-28]] |  [[https://datatracker.ietf.org/doc/draft-ietf-rtcweb-jsep|draft-ietf-rtcweb-jsep]] | 
 +| Pages submitted |  156 pages |   115 pages | 
 +| I-D approved    2018-03-21 |   2018-03-01\\ (into MISSREF state) | 
 +| AUTH48 start    2018-06-14 |   2020-07-06 | 
 +| Publication    |  2018-08-10 |   2021-01-20 | 
 +| GitHub repo    |  https://github.com/tlswg/tls13-rfc |  https://github.com/rtcweb-wg/jsep | 
 +^ AUTH48 details  ^^^ 
 +| Time in state  |   8.2 weeks |   28.3 weeks | 
 +| # questions at start  |   58 |    60\\ (yielded [[https://github.com/rtcweb-wg/jsep/issues?q=is%3Aissue+is%3Aclosed+assignee%3Aajeanmahoney|78 issues]] in GitHub) |   
 + 
 +==== Process Plan in 2018 ====
  
   - The author will create a repository with just the XML. We ask that the author not use their existing repository, for various reasons.   - The author will create a repository with just the XML. We ask that the author not use their existing repository, for various reasons.
Line 9: Line 22:
     * status of this memo     * status of this memo
     * section movement to meet the style guide requirements (e.g., move acks and contribs to appear at the end)     * section movement to meet the style guide requirements (e.g., move acks and contribs to appear at the end)
-  - RFC Editor goes through the EDIT and RFC EDITOR process states and sends an email with a link to the XML file created in step 2 and the edited XML file at the start of AUTH48.+  - RFC Editor goes through the EDIT and RFC EDITOR process states and sends an email with a link to the XML file created in step 2 and the edited XML file at the start of AUTH48. [Both files provided 2018-06-14]
   - At the start of AUTH48, the author will create a Pull Request (PR) with the changes proposed by the RPC. The authors respond and point to the repository they created.   - At the start of AUTH48, the author will create a Pull Request (PR) with the changes proposed by the RPC. The authors respond and point to the repository they created.
   - The relevant people from the RPC will subscribe to notifications from that repository using the “Watch” button and confirm they are watching. For this test case, Heather, Adam, and Martin will follow along as well.   - The relevant people from the RPC will subscribe to notifications from that repository using the “Watch” button and confirm they are watching. For this test case, Heather, Adam, and Martin will follow along as well.
Line 21: Line 34:
   - Final approval for the full document needs to result in an explicit email to the the RPC from each author indicating approval. GitHub will be used for editorial discussion, but the final text approval will be done via email.    - Final approval for the full document needs to result in an explicit email to the the RPC from each author indicating approval. GitHub will be used for editorial discussion, but the final text approval will be done via email. 
  
-=== Relevant Documents ===+==== Evaluation Criteria in 2018 ==== 
 + 
 +=== RPC Criteria === 
 + 
 +Does GitHub seem to provide an easy-to-use mechanism for: 
 +  * tracking changes during AUTH48 in such a way that the RPC can query, at any point of time in the future, who approved those changes? 
 +  * having clear interactions with all parties that need to submit approvals during AUTH48? 
 +  * a reasonable (no longer than bringing a new editor up to speed on current AUTH48 processes) learning curve for the use of GitHub?  
 + 
 +=== Measurements === 
 +  * Average time in AUTH48 state for standard process documents 
 +  * Average time in AUTH48 state for 100+ page standard process documents 
 +  * Time in AUTH48 state for this document  
 +  * Number of questions at start of AUTH48 
 +  * Number of pull requests afterwards 
 + 
 +=== Measurements in 2018 and 2020 === 
 +^                            ^   RFC 8446   ^ RFC 8829   ^ 
 +| AUTH48 time using GitHub   | 8.2 weeks     | 28.3 weeks    | 
 +^ Comparing with standard process ^^^ 
 +| Concurrent Avg. AUTH48 time*   | 3.2 weeks    | 10.1 weeks    | 
 +| Avg. AUTH48 time for docs over 100 pages<nowiki>**</nowiki>  | 4.4 weeks (n=7)  | 3.5 weeks (n=6)   | 
 + 
 + 
 +* using standard AUTH48 process for the 3 months preceding publication.\\ 
 +<nowiki>**</nowiki> using standard AUTH48 process for the 18 months preceding publication. 
 + 
 + 
 +=== Notes === 
 +  * future work will require mapping of GitHub account names to Authors, WG Chairs, Document Shepherds, ADs, Stream Managers (possible GDPR implications this information is outside the publishing industry norm for information stored about an individual during a publication process) 
 + 
 +==== Feedback in 2018 ==== 
 + 
 +  * See [[github_exp_2018_feedback]] for notes from the author and the RFC Editor. 
 + 
 +-------------------------------- 
 + 
 +==== Work Flow in 2020-2021 ==== 
 + 
 +The authors and editor agreed to use the GitHub Pull Request work flow:
  
-  * draft-ietf-tls-tls13-28  +  - The authors answer the AQ in the comments of the issue (e.g., see comments for [[https://github.com/rtcweb-wg/jsep/issues/922|#922]]). 
-    * 2018-03-21: added to the queue+  The authors indicate that an issue is ready for editor attention by labeling the issue "editor-ready" and assigning the issue to the editor. 
-  * draft-ietf-rtcweb-jsep-24 +  The editor creates a branch off the rfced branch to address the issue
-    * 2018-03-01: added to the queue in MISSREF because of unapproved normative reference.+  - The editor makes edits in that branch. 
 +  The editor commits changes. 
 +  - The editor creates a Pull Request (PR) to submit the changes to the repo (e.g., [[https://github.com/rtcweb-wg/jsep/pull/997|PR #997]]). 
 +  The authors review the PR:  
 +    - If the PR is accepted, the authors merge the changes to the rfced branch, which closes the issue. 
 +    If the PR is rejected, the editor makes changes to the current branch and resubmits the PR.  
 +  - Repeat for the other issues.   
 +   
 +The authors and the editor set up notifications so that many of these steps automatically generated emails to involved parties (assigning/commenting on issues, use of @mentions, creating/accepting/rejecting PRs). 766 messages were generated during AUTH48.
  
-=== Evaluation Criteria ===+==== Feedback in 2020-2021 ====
  
-To be added.+  * See [[github_exp_2021_feedback]] for notes from the RFC Editor.
github_auth48_experiment.1528912398.txt.gz · Last modified: 2018/06/13 17:53 by arusso