Sunday, February 26, 2012

C++ Iterators Like Python Generators

I just showed a colleague some Python written in the style of Dave Beazley's generators for system programming, and he wondered whether we could use something similar in C++. Copying Python's syntax would be tortuous in C++, but I think we can match the spirit.

Instead of loading a logfile in Python and creating a new copy of it every time the code transforms it, code written as generators works through the logfile line-by-line, sparing memory while separating transformations clearly into separate code segments. Consider a short example:


import re
def matches_numeric(file_lines):
for line in file_lines:
if re.match('^[0-9 \t\.]$',line):
yield line


if __name__ == '__main__':
file_handle=open('infile.txt','r')
data=matches_numeric(file_handle)
columnar=split_columns(data)
checked=basic_checks(columnar)
red_flags=search_error(checked)
for line in red_flags:
print line

This way, the transformations are represented clearly and are easy to mix and match.

C++ doesn't have a yield keyword. It's iterators don't signal that they are complete by throwing StopIteration. Instead the iterator has to match an end-of-stream iterator, so that's what we can construct in C++. This means that the moral, but not syntactic, equivalent of Python generators is a function that returns a pair of iterators.

template<class SOURCE>

boost::array<split_iterator< SOURCE >,2> split_line(boost::array< SOURCE,2>& begin_end) {
boost::array<split_iterator< SOURCE >,2> iters = {{
split_iterator< SOURCE >(begin_end), split_iterator< SOURCE >()
}};
return iters;
}

These iterators are packages in a boost::array, but you could use a std::pair, or not package them, as you please, but the goal is the same, to create a nice way to express a series of transformations.

std::ifstream in_file("z.txt");
auto file_line=file_by_line(in_file);
auto splits=split_line(file_line);
while (splits[0]!=splits[1]) {
for (auto word=begin(*splitted); word!=end(splitted); word++) {
std::cout << *word << ":";
}
std::cout << std::endl;
}

The C++ looks similar to the Python, but each transformation is building on the type of the previous transformation, so it ends up doing type chaining in a less explicit way than boost::accumulators.

The code is on github.

Friday, January 13, 2012

A Quick Check for C++11 Features

The new C++11 features are exciting but I need to know which features my various compilers support, so I wrote an SCons script that tries to compile samples of new features. Seeing the samples compile is quicker, and more understandable to me, than looking up Intel's C++11 list or GCC C++0x support. Plus, I noticed that Intel's online list has almost all Yes's, but they don't list some of the Wikipedia entries that would be a No.

The SCons script, called Cpp11check, is on Github. Edit local.cfg to specify your compiler. Then run "scons" to see a summary or "scons --echo" to see every test snippet it compiles.

Looking at sample output from Intel's C++ 12.1, it looks to me like they concentrated on language features and have yet to include in the std namespace functionality that is in Boost. Seems like a decent choice. I just wish initializer lists worked. {{I, love, those, things.}}

-bash-3.2$ scons
scons: Reading SConscript files ...
INFO:SconsRoot:running with 2 threads
ERROR:SconsRoot:Could not find g++ with a version.
INFO:SconsRoot:Testing C++ compiler: /opt/intel/composer_xe_2011_sp1.6.233/bin/intel64/icpc
Checking whether the C++ compiler worksyes
Checking for c++0x conformance...-std=c++11?...-std=c++0x?...yes
Checking snippet alias templates...yes
Checking snippet alternative function syntax...yes
Checking snippet explicit final...no
Checking snippet explicit override...no
Checking snippet explicitly defaulted special member functions...yes
Checking snippet explicitly deleted member functions...yes
Checking snippet generalized_constant...no
Checking snippet hash tables...no
Checking snippet initializer lists...no
Checking snippet lambda functions...yes
Checking snippet long long int...yes
Checking snippet new string literals...no
Checking snippet nullptr...yes
Checking snippet object construction constructors calling constructors...no
Checking snippet object construction improvement using base constructor...no
Checking snippet polymorphic wrappers for function objects...no
Checking snippet random numbers...no
Checking snippet range-based for-loop...no
Checking snippet regex...no
Checking snippet right angle brackets...yes
Checking snippet shared_ptr...no
Checking snippet sizeof on member objects...yes
Checking snippet static_assert...yes
Checking snippet strongly-typed enum...yes
Checking snippet templates with variable number of values...yes
Checking snippet tuple...no
Checking snippet type inference auto...yes
Checking snippet type inference decltype...yes
Checking snippet type traits metaprogramming...no
Checking snippet uniform initialization...no
Checking snippet unrestricted unions...no
Checking snippet user-defined literals...no
Checking snippet using syntax instead of typedefs...yes
Checking snippet wrapper reference...no
scons: done reading SConscript files.
scons: Building targets ...
scons: `.' is up to date.
scons: done building targets.
Build succeeded.

HTH,
Drew