TODO.txt

Upgrade to LLVM 3.6:
    remove dependencies on JITMemoryManager
    resolve changes to NotifyObjectEmitted
    resolve changes to LinkModule (need to copy the src module, fix error handling)
    resolve changes to Metadata types all over the code

Debugging:
  terralib.traceback() -- make a version of debug.traceback() that is aware of Terra compilation state. Rather than printing the raw stack frames for specialization/compilation, it should report the line
  of Terra code that it is doing the operation:
  
    foo.t:42: assertion failed!
    foo.t:39 specializing
            var a = [b + c]
                    ^
    ...
terralib.totype() -- rather than use istype, have a totype that returns nil on failure. Replace istype checks with totype. Allow {int,int} convert, and we can consider a toterratype check in an objects meta table as well.

remove initialization of cuda runtime. Add cudalib.linkcuda() to load the cuda runtime (this will speed startup time for things that do not use cuda).

Should exotypes be objects rather than modifying a table? Or should we have toterratype? Both (seems like a bad idea to have both). If exotypes are objects, they should be able
to do almost everything except the things that the original type has to do.
  

Cross-compilation:


-- store target, method to allocate a target machine (used for our own target machine and for EngineBuilder::create)
-- use target machine to make JIT rather than separate config path
-- allow createcompilationunit to take arguments: triple, cpu, arch, featurestr specified optionally, "native" will mean to use LLVM to discover the native values
-- each function that uses createccompilationunit will take the arch optionally
-- make sure X86_64 assumptions are not in other places
-- stringify and pass correct arguments to clang for cross compilation as well


Goal: Make it really easy to emit C header files for Terra functions

-- API to get list of parameter names of a function, rather than just the type
-- edit the implementation of cstring so that it can be used externally as well as internally


Language Implementation:

-- Represent aggregate constants as aggregates rather than current method of casting
-- switch construct, computed goto
-- vector shuffle?
-- cmpxchg, atomicrmw, fence (all but fence can be part of attrstore)

Design Problems:
-- how to save results for later (both Lua state and Terra JIT state)

Low-priority Implementation:
-- generate TBAA metadata like C will so that we can match perf (do we need?)
-- gather and scatter for vectors (allow vec(int*) and its loads)
-- pointer arithmetic on niltype?
-- better handling of options to try for overloaded operators


DONE:

--function pointers
--anonymous structural type cast to other structural types
--methods on types (declaration syntax )
--methods on types (call syntax/resolution) 
-- -> and & syntax need to work in normal lua code (not just type resolution)
-- types should be presented in the AST as the string of lua code to run to get the type rather than an AST needed to be interpreted--eval types:
   change types to have either canonical versions or exist as proxy objects
   type constructors (e.g. pointer to) should ask if a type is canonical and create a proxy if not)
--pointer arithmetic
--array types
--resolvetype needs to allow the declaration of anonymous structural types
--macros in terra
-- explicit casts (pointer to pointer, requires macros to somewhat work)
-- constructor syntax with a function in last position should create 1 anonymous entry per return value of the function
-- string literals translated to int8* types
-- how to use clang to read C header files
-- an opaque type (equivalent to void* in C) (? another alternative is to just use int8* for  now) (went with: no unsized opaque types, but some imported c types do now expose internals)
-- support typedefs and record types in C header files
-- automatic conversion from any pointer to &uint8
-- nil should translate to any pointer type's NULL value, this will require adding an implicit conversion for <nulltype> to any pointer type,
   and allowing nulltype to exist
-- union annontations in struct types
-- support for aggregate types in Lua-terra interface
   extend cstring to work for arrays and structs (+unions) being careful to work for recursive types
   to work around c's weird syntax, each non-primitive will need to generate a typedef for its type
   fold the typedef for functions into cstring and simplify makewrapper to use its type instead
   pointers to functions will need to be handled as a special case
--calling lua functions from terra
-- turn debug on/off debug printing
-- introduce LL, ULL, and f to make constants have the expected types
-- simple save of functions (give a key-value pair and export the symbols with those names to a .o file, see if you can use aliases to export these functions)
-- code quotations of statements and expressions
-- getting started guide
-- revise how quoted code is injected into AST. In particular, macros should not drop Quote objects into the AST (it should instead resolve the quotes in place)
   also consider the way that checkexp allows quote objects to propogate: consider a different design where untyped code is first resolved:
      vars are check if they are Quotes and inserted as Quote nodes
      same goes for global variables
      partial eval of select is handeld
      macro nodes are expanded (recursively) until there are no non-tree literals left
-- good error message for when there is a circular dependency on global variable initializationlocation like old C for now)
-- getting the right -O3 passes (llvm-as < /dev/null | opt -O3 -disable-output -debug-pass=Arguments)
   handle as much as possible in the JIT, and make sure the offline compiler does everything
   make configurable (perhaps a general config mechanism) (look a preconfigured stuff)
-- write 1 or two language benchmarks in terra to get baseline perf (do this after figuring out -O3)
-- creating references to variables found in another quote object despite hygiene
-- vector types
-- overloaded functions
-- overloaded operator behavior
-- fix gc issues cleaning up functions: 1. either resolve symbols for quotes and functions once and throw away the env, or 2. trace references to local variables in the parser and construct the environment so that it only captures the referenced variables (this is probably the better solution, but changing the parser to do it may be tricky).
-- returning values from lua functions (via terralib.cast)
-- better interaction for low-level values allocated from Lua and used in Terra. (terralib.constant behavior implemented for ctypes)
--&&type should convert to &&uint8 (check number of addresses is the same, currently only allows &foo to convert to &uint8) (did not fix, this isn't how C behaves either)
--look into bug: if you cast an lvalue pointer to another pointer, it becomes an rvalue, this is not right... (did not fix, that is the right behavior)
-- change parser rule to allow let rec form that can include  local/global structs/terra declarations/definitions together. modify terralib to accept this form and remove constructors for other forms.
-- tag _all_ tree nodes with the filename and remove the complicated stack of files for error reporting (we can compress tree node size later)
-- remove var syntax from lua. Instead use terralib.global to create global variables.

Implementation:
-- better handling of lua layout of structs with recursion (can we seperate the creation of structs for their layout to simplify the problem) -- test the compiler fo handling jitting of functions previously optimized with nojit = true
   (look into the way LLVM handles pointers to opaque structs, can we delay layout of a struct until we need to access it similar to the way C allows opaque struct pointers).

-- tests for garbage collection of objects that mix LLVM state with Lua state to check for memory leaks from dangling references

--track down why some types RecordType structs do not have names, and fix the bugs that are caused by this. 

--refactor ctx object and terra.resolvetype it should be possible to canonicalize and resolve types without a ctx object
  instead there should be a seperate diagnostic object for reporting errors, and resolvetype should be given an environment in which to run
  getcanonical() should then take an _optional_ diagnostic object

-- casting lua functions to Terra functions when they return multiple arguments, and supporting calling Lua functions with structs values/expecting struct returns

Making specialization eager and simplifying the compiler.

-- seperate specialization phase (e.g. variable resolution, select resolution, escape evaluation) from typechecking, specialization will be run eagerly

-- make the compilation graph a singleton, environments should be handled by function compilation individually. If something is re-entrant into compilation from lua code and there are compile errors stop compilation. Double-check that re-entrant compiler code can compile as long as it is in a different SCC from its compile-time caller. Remove ctx objects as arguments. fnvariant:gettype() and fnvariant:compile() should be seperate from the internal interfaces.

-- add asynchronous versions of :compile and :gettype that can be called when the compiler is running
-- support LLVM constant expressions: terralib.constant(`exp) (this will allow vtables etc. to be constructed)

-- (lower priority) allow codegeneration for opaque struct types when they are only used as pointers. this will require cooperation from the compiler to keep the struct opaque unless it is marked complete. compiler will need to be annotated to force completion of struct types when they are used as values (either directly or in an aggregate).

Minitasks that came up during eager transform:

--luafunction is gone
--write the "documentor"
--the attribute macro is broken
-- getcanonical should cache whether it has been done on aggregates so it doesn't traverse types a bunch of times

Things to write tests/audit for:

-- literals are only generated by the parser, constants handles all stuff from lua (including strings), add correct support for strings
-- select and var handling have moved
-- truncation behavior (since it was changed from a field to a node)

-- pretty print is being weird on dgemmpaper.t, add support for typedexpressionlist
Goal: switch from one emission of llvm to architecture-specific version of llvm.

-- Add a compiler intermediate compilationunit, which maps function definition, global, constant, and everything that has a llvm_value field right now to the llvm value. 

-- Remove emitllvm, typechecking now only handles generating types and invoking user callbacks, never emitting/optimizing llvm

-- consider refactoring the SCC detection into it own object, or labelling each function with an identifier for the SCC it is in

-- switch back to SCC-first order optimization for MCJIT

--  simplify the state for functions: untyped, in progress, typed, re-examine peektype, etc.
--  see if extern functions do not need "initialization" anymore

-- emitting llvm is now given a compilationunit, a root function to add to it, and archiecture
-- include C creates a separate compilationunit, examine linking rules for including it.