Introduction

The idea behind this project is that 2to3 fixers might attempt to apply certain heuristics to translation that may not be guaranteed to work, but which should work up to a certain confidence level, given the context. Users would be able to inspect changes in which fixers have less confidence and decide what to do for each situation: apply the fix, skip it or edit original the code and try to re-apply the fix. A variation of the latter action would be to manually edit the resulting 3.0 code and use that instead of the fix.

For example, the fix_has_key fixer might rank the translation of a has_key() method being called on an instance of a class not related to dict with less confidence than it would on a dict instance. In this situation, if a sufficiently low confidence threshold has been set, the proposed change (i.e., "obj.has_key(k)" being translated into "k in obj") would be shown to the user and he might choose to skip it if obj.has_key is not related to finding dictionary keys.

2to3 New Behaviour

When running 2to3, the user will have a new command-line option, --confidence-threshold, indicating a confidence level below which fixes should not be applied automatically. There will also be a --default-confidence option, set by default to be immediately below the confidence threshold, for fixers that do not implement ranking.

During 2to3's run, each fixer, when given a node to transform, will return the new node and the confidence ranking of the change. The specifics of how fixers will rank their translations, of course, are unique to each fixer, and this project includes studying all of them to determine how (and if) different ranking heuristics might be applied.

2to3 will then, based on the --confidence-threshold option, determine whether it is necessary to ask the user to intervene. If it is not necessary to have user intervention, 2to3 will move on as it currently does.

When a fix is ranked below the confidence threshold, the user will be presented with the proposed changes. This, in its simplest form, is a GNU diff-like output comparing the original code and the translated one. The user will also be able to specify, via another command-line option, an external program that receives two input files and compares them in a similar fashion, enabling the use of graphical tools such as Meld (http://meld.sourceforge.net/).

Once presented with the fix, the user is given four choices:

Apply

2to3 replaces the original node with the transformed one and moves on to the next fix or node.

Skip

2to3 leaves the node unaltered by the current fix, and moves on to the next fix or node.

Edit original code

The code corresponding to the original node is presented inside a text editor to the user, who may change it at will (given that all changes must remain within that same node). Once the user quits the editor, the node is re-submitted to the fixer and, if it still matches, new fix and ranking are provided and a new decision on whether to ask the user for intervention is made.

Edit transformed code

The code corresponding to the transformed node is presented inside a text editor to the user, who may change it at will (given that all changes must remain within that same node). Once the user quits the editor, the changes are applied as the new node, and 2to3 moves on to the next fix or node.

After the user has chosen one of the options, 2to3 responds as described above and carries on with its regular execution.

Testing

The 2to3 tool will be tested, as the new behaviour is implemented, both on code created specifically for testing and on existing open source projects. Among these, there will a substantial amount of tests on the Django project's source code. Django is complex enough that the tests will be significant and likely show interesting corner cases. The Django project itself might benefit from a code base that can be run with Python 3.0, although that is not the main goal of this proposal.

From these tests, not only will 2to3 bugs be corrected, but it will be possible to determine reasonable default confidence and threshold values.

The Django project may be found at http://www.djangoproject.com/ . Tests with 2to3 and Django will build upon the experiment conducted by Martin Löwis at http://wiki.python.org/moin/PortingDjangoTo3k .

Proposed Timeline

These steps assume updating documentation at each behaviour change, and continuous testing as described previously.

• April 14 - May 26
  • Define communication channels (instant messaging, irc, lists etc) and a work environment (bzr repositories, desktop-sharing tools etc) with mentor.
  • Define a work schedule (meetings, checkpoints etc) with mentor.
  • Review Python-Dev guidelines and tools.
  • Get more acquainted with 2to3 code.
  • Define specific behaviour of proposed changes (option names, ranking style etc).
• May 26 - June 2
  • Add new command-line options.
  • Modify 2to3 to work with the default case (no fixers know about ranking, 2to3 should respect --confidence-threshold and --default-confidence).
• June 2 - June 16
  • Add interactive mode to show GNU diff-like output and get user choice.
  • Add handling of each of the possible choices.
  • Add new command-line options to specify external diff program.
  • Test with popular external diff programs, including graphical ones.
• June 16 - July 28
  • For each of the (currently) 40 fixers, implement ranking based on each of their specific heuristics.
  • July 7: mid-term evaluations.
• July 28 - August 11
  • Determine and document default confidence and threshold values from tests with existing open source projects, as described previously.
  • Ideally, verify merging status with other 2to3-related Google Summer of Code projects and help them adjust to ranking, if needed.
• August 11 - August 18
  • Final tests.
  • Final code refactoring, if needed.
  • Update and verify documentation.
• August 18
  • Google Summer of Code deadline

Updates

This project proposal and updates to it are available at http://isnomore.net/2to3