tag:blogger.com,1999:blog-244899062024-03-07T19:19:24.575-05:00Computing KitchenScientific simulation and visualization for developers.Drew Dolgerthttp://www.blogger.com/profile/10755676286500857195noreply@blogger.comBlogger21125tag:blogger.com,1999:blog-24489906.post-92030633647229063672012-02-26T16:20:00.000-05:002012-02-26T16:27:17.182-05:00C++ Iterators Like Python GeneratorsI just showed a colleague some Python written in the style of Dave Beazley's <a href="http://www.dabeaz.com/generators/">generators for system programming</a>, and he wondered whether we could use something similar in C++. Copying Python's syntax would be tortuous in C++, but I think we can match the spirit.<br />
<br />
Instead of loading a logfile in Python and creating a new copy of it every time the code transforms it, code written as generators works through the logfile line-by-line, sparing memory while separating transformations clearly into separate code segments. Consider a short example:<br />
<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">import re</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">def matches_numeric(file_lines):</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>for line in file_lines:</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>if re.match('^[0-9 \t\.]$',line):</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>yield line</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">if __name__ == '__main__':</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>file_handle=open('infile.txt','r')</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>data=matches_numeric(file_handle)</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>columnar=split_columns(data)</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>checked=basic_checks(columnar)</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>red_flags=search_error(checked)</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>for line in red_flags:</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>print line</span><br />
<br />
This way, the transformations are represented clearly and are easy to mix and match.<br />
<br />
C++ doesn't have a yield keyword. It's iterators don't signal that they are complete by throwing StopIteration. Instead the iterator has to match an end-of-stream iterator, so that's what we can construct in C++. This means that the moral, but not syntactic, equivalent of Python generators is a function that returns a pair of iterators.<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">template<class SOURCE></span><br />
<br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">boost::array<split_iterator< SOURCE >,2> split_line(boost::array< SOURCE,2>& begin_end) {</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>boost::array<split_iterator< SOURCE >,2> iters = {{</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>split_iterator< SOURCE >(begin_end), split_iterator< SOURCE >()</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>}};</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>return iters;</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">}</span><br />
<div>
<br /></div>
<div>
These iterators are packages in a boost::array, but you could use a std::pair, or not package them, as you please, but the goal is the same, to create a nice way to express a series of transformations.</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">std::ifstream in_file("z.txt");</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">auto file_line=file_by_line(in_file);</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">auto splits=split_line(file_line);</span></div>
</div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">while (splits[0]!=splits[1]) {</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>for (auto word=begin(*splitted); word!=end(splitted); word++) {</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>std::cout << *word << ":";</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>}</span></div>
<div>
<span class="Apple-tab-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small; white-space: pre;"> </span><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">std::cout << std::endl;</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">}</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span></div>
<div>
The C++ looks similar to the Python, but each transformation is building on the type of the previous transformation, so it ends up doing type chaining in a less explicit way than boost::accumulators.<br />
<br />
The code is on <a href="https://github.com/adolgert/cpp_generators">github</a>.</div>
<div>
<br /></div>Drew Dolgerthttp://www.blogger.com/profile/10755676286500857195noreply@blogger.com0tag:blogger.com,1999:blog-24489906.post-40643092097642608772012-01-13T15:52:00.000-05:002012-01-13T15:52:46.054-05:00A Quick Check for C++11 FeaturesThe new <a href="http://en.wikipedia.org/wiki/C++11">C++11</a> features are exciting but I need to
know which features my various compilers support, so I wrote an <a href="http://www.scons.org/">SCons</a>
script that tries to compile samples of new features. Seeing the samples compile is quicker, and
more understandable to me, than looking up <a href="http://software.intel.com/en-us/articles/c0x-features-supported-by-intel-c-compiler/">Intel's C++11 list</a> or <a href="http://gcc.gnu.org/projects/cxx0x.html">GCC C++0x support</a>. Plus, I noticed that
Intel's online list has almost all Yes's, but they don't list some of the Wikipedia entries that would be a No.
<br />
<br />
The SCons script, called <a href="https://github.com/adolgert/Cpp11check">Cpp11check</a>, is on
Github. Edit local.cfg to specify your compiler. Then run "scons" to see a summary or "scons --echo"
to see every test snippet it compiles.
<br />
<br />
Looking at sample output from Intel's C++ 12.1, it looks to me like they concentrated on language features and have yet to include in the std namespace functionality that is in Boost. Seems like a decent choice. I just wish initializer lists worked. {{I, love, those, things.}}<br />
<br />
<pre>-bash-3.2$ scons
scons: Reading SConscript files ...
INFO:SconsRoot:running with 2 threads
ERROR:SconsRoot:Could not find g++ with a version.
INFO:SconsRoot:Testing C++ compiler: /opt/intel/composer_xe_2011_sp1.6.233/bin/intel64/icpc
Checking whether the C++ compiler worksyes
Checking for c++0x conformance...-std=c++11?...-std=c++0x?...yes
Checking snippet alias templates...yes
Checking snippet alternative function syntax...yes
Checking snippet explicit final...no
Checking snippet explicit override...no
Checking snippet explicitly defaulted special member functions...yes
Checking snippet explicitly deleted member functions...yes
Checking snippet generalized_constant...no
Checking snippet hash tables...no
Checking snippet initializer lists...no
Checking snippet lambda functions...yes
Checking snippet long long int...yes
Checking snippet new string literals...no
Checking snippet nullptr...yes
Checking snippet object construction constructors calling constructors...no
Checking snippet object construction improvement using base constructor...no
Checking snippet polymorphic wrappers for function objects...no
Checking snippet random numbers...no
Checking snippet range-based for-loop...no
Checking snippet regex...no
Checking snippet right angle brackets...yes
Checking snippet shared_ptr...no
Checking snippet sizeof on member objects...yes
Checking snippet static_assert...yes
Checking snippet strongly-typed enum...yes
Checking snippet templates with variable number of values...yes
Checking snippet tuple...no
Checking snippet type inference auto...yes
Checking snippet type inference decltype...yes
Checking snippet type traits metaprogramming...no
Checking snippet uniform initialization...no
Checking snippet unrestricted unions...no
Checking snippet user-defined literals...no
Checking snippet using syntax instead of typedefs...yes
Checking snippet wrapper reference...no
scons: done reading SConscript files.
scons: Building targets ...
scons: `.' is up to date.
scons: done building targets.
Build succeeded.
</pre>
<br />
HTH,<br />
DrewDrew Dolgerthttp://www.blogger.com/profile/10755676286500857195noreply@blogger.com0tag:blogger.com,1999:blog-24489906.post-83402144639980164622011-10-03T09:26:00.002-04:002011-10-03T09:59:13.409-04:00Installation of Boost.Python on Mac OS XWith the current MacPorts version of Boost 1.47.0, I can't follow the <a href="http://www.boost.org/doc/libs/1_41_0/libs/python/doc/building.html">Boost.Python installation instructions.</a> I installed the three relevant ports, boost +python27, boost-build, and boost-jam. The installation instructions recommend using the python/quickstart directory, and the include paths in the Jamfile don't exist. The Jamfile in the port is even missing the "import python" statement necessary to load the python-extension rule. The lesson is that it's OK to give up on a MacPorts installation when you are using an unusual feature.<br /><br />Install boost-1.47.0 from source. Given that there is already a MacPorts installation, whose default install directory is /opt/local, put the boost_1_47_0 directory directly into /opt as /opt/boost_1_47_0. Follow build instructions to make boost's bjam and b2, so that the whole lot end up in /opt/bin, /opt/include, and /opt/lib. My build command was:<br /><br />sudo ./bootstrap.sh --with-bjam=/opt/local/bin/bjam --with-toolset=darwin --with-python=/opt/local/bin/python2.7 --prefix=/opt --without-libraries=mpi,regex<br /><br />I was trying to use the MacPorts bjam, but Boost built its own, anyway, which turns out to be good because it builds the newer b2 version of bjam. Boost.Python defaults to the Mac OS X default Python, so why not specify your favorite version? Then return to the Boost.Python installation instructions. I had to set the path so that the newer Boost is earlier:<br /><br />export DYLD_LIBRARY_PATH=/opt/lib<br />export PATH=/opt/bin:$PATH<br /><br />There are still going to be errors about conflicts with isspace and other functions in localfwd.h. These come from a conflict with newer definitions in pyports.h that are designed to handle UTF-8. I got around these by installing MacPorts port for gcc45. Then, in the Jamroot of the sample directory, add:<br /><br />using darwin : 4.5.3 : g++-mp-4.5 ;<br /><br />The darwin import is derived from the gcc import, so you can give it pretty much the same options. In this case, it points directly to g++. Once this is done, you can run "sudo bjam" in the quickstart directory to build Boost.Python. This will build the libboost_python library so you can now run without using sudo in your project's working directory.<br /><br />Still not done. If you are using Numpy arrays in Boost.Python, then you need to include the correct headers, meaning '#include "numpy/arrayobject.h"'. These are installed in a separate place on my Mac, again likely by MacPorts, but can be found by a change to the python-extension rule.<br /><br />python-extension myproject : file.cpp<br /> : <include>/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include ;<br /><br />How do you find these things? Use the Mac's whole-machine find command on the command-line.<br /><br />mdfind -name arrayobject.h -onlyin /opt<br /><br />In the end, the boost-build.jam contains "boost-build /opt/boost_1_47_0/tools/build/v2 ;" and the project Jamroot uses "use-project boost : /opt/boost_1_47_0".<br /><br />HTH,<br />DrewDrew Dolgerthttp://www.blogger.com/profile/10755676286500857195noreply@blogger.com0tag:blogger.com,1999:blog-24489906.post-37134438657773558332011-03-09T10:29:00.002-05:002011-03-09T10:34:46.576-05:00skipping incompatible in /usr/lib64I was building a 32-bit executable on a 64-bit machine, using the -m32 switch, and thought I knew how to deal with this error from the g++ or gcc compiler.<br /><br />/usr/bin/ld: skipping incompatible /usr/lib64/libelf.so when searching for -lelf<br />/usr/bin/ld: skipping incompatible /usr/lib64/libelf.a when searching for -lelf<br />/usr/bin/ld: cannot find -lelf<br /><br />I checked the link line for any binaries that might be 64-bit by running the file command, as in "file tau_run.o". They all looked 32-bit. I checked my LIBRARY_PATH environment variable, which tells the linker in which directories to find libraries.<br /><br />The problem turned out to be not that the compiler was looking in the wrong <span style="font-style:italic;">place</span> but that it was looking for the a file that did not exist. The dyninstAPI library has a file called libelf.so.1, but they did not include a link from libelf.so.1 to libelf.so, which is what the compiler wants to find. The compiler was doing a great job of rejecting 64-bit libraries and its error message was actually rather clear that it could not find the -lelf it was looking for.<br /><br />As usual, HTH.<br />DrewDrew Dolgerthttp://www.blogger.com/profile/10755676286500857195noreply@blogger.com0tag:blogger.com,1999:blog-24489906.post-79959384481651298822011-02-08T22:42:00.002-05:002011-02-08T22:50:44.753-05:00How Does R Build a Package with C Source Code?<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; background-color: transparent; "><h2 id="internal-source-marker_0.5006632145959884" style="font-size: medium; "><span class="Apple-style-span" style="font-weight: normal; "><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">According to </span><a href="http://cran.r-project.org/doc/manuals/R-exts.html" style="font-family: Times; "><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 153); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap; ">Writing R Extensions</span></a><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">, I put C files into a package subdirectory and supply a configure.ac</span><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; "> and Makevars.in, and the command “R CMD INSTALL” will compile the extensions. I’m working in Linux, so it uses configure somehow, but what exactly does it do? Where do the defaults come from and how can I change them? As of R version 2.12.1, the build scripts are in the R tools package, no longer written in Perl.</span></span></h2><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">There are a lot of files to put in an R package, from help files to a general description of the library. We are concerned with these three, as an example, for a library called foo.</span><br /><span style="font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; font-size: 11pt; background-color: transparent; font-size: 11pt; background-color: transparent;"></span><ul style="font-family: Times; font-size: medium; "><li style="list-style-type: disc; font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; "><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">foo/configure.ac</span><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; "> - Input to create a configure script.</span></li><li style="list-style-type: disc; font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; "><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">foo/src/Makevars.in - The configure script will create Makevars from this file.</span></li><li style="list-style-type: disc; font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; "><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">foo/src/foo.c - This is our C source to compile.</span></li></ul><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">There are three commands R makes available to work with source while you develop it.</span><br /><span style="font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; font-size: 11pt; background-color: transparent; font-size: 11pt; background-color: transparent;"></span><ul style="font-family: Times; font-size: medium; "><li style="list-style-type: disc; font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; "><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R CMD build foo - This creates a tar file of the directory suitable for installing later.</span></li><li style="list-style-type: disc; font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; "><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R CMD INSTALL foo - This compiles and installs foo into a directory.</span></li><li style="list-style-type: disc; font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; "><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R CMD check foo - This does all kinds of detailed checks on the health of the R package, all listed in Writing R Extensions, and it also calls R CMD install, so it’s a good way to smoke test compilation.</span></li></ul><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">If you call “R CMD check foo”, then it calls the library tools:::.check_packages() which eventually installs the library into a local directory by calling R again:</span><br /><span style="font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; font-size: 11pt; background-color: transparent; font-size: 11pt; background-color: transparent;"></span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R CMD INSTALL -l '/home/username/Documents/rlib/foo.Rcheck' --no-html --no-multiarch '/Users/ajd27/Documents/rlib/foo'</span><br /><span style="font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; font-size: 11pt; background-color: transparent; font-size: 11pt; background-color: transparent;"></span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">As an aside, calling “R CMD blah” sets R environment variables and then invokes a shell script which looks for a script called blah in R’s bin directory. If it doesn’t find one, it just executes whatever you passed it. Try “R CMD ls -la .” or “R CMD env|sort” to see what environment variables R defines.</span><br /><span style="font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; font-size: 11pt; background-color: transparent; font-size: 11pt; background-color: transparent;"></span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">The install command is implemented in tools:::.install_packages(). The easiest way to see what it does is to look in the R source code, in the src/library/tools/R/install.R. It executes these steps on your behalf.</span><br /><span style="font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; font-size: 11pt; background-color: transparent; font-size: 11pt; background-color: transparent;"></span><ol style="font-family: Times; font-size: medium; "><li style="list-style-type: decimal; font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; "><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">Define R-specific variables. These are listed below for one sample.</span></li><li style="list-style-type: decimal; font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; "><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">Call autoconf to create foo/configure from foo/configure.ac.</span></li><li style="list-style-type: decimal; font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; "><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">Call foo/configure, whose main goal is to make foo/src/Makevars from foo/src/Makevars.in.</span></li><li style="list-style-type: decimal; font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; "><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">Look for makefiles in foo/src and call make in foo/src to create shared libraries.</span></li></ol><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">We can specify arguments to configure on the INSTALL command line, with --configure-args and --configure-vars. For instance, typing</span><br /><span style="font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; font-size: 11pt; background-color: transparent; font-size: 11pt; background-color: transparent;"></span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R CMD INSTALL “--configure-args=--enable-lizards --disable-frogs”</span><br /><span style="font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; font-size: 11pt; background-color: transparent; font-size: 11pt; background-color: transparent;"></span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">will call</span><br /><span style="font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; font-size: 11pt; background-color: transparent; font-size: 11pt; background-color: transparent;"></span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">./configure --enable-lizards --disable-frogs</span><br /><span style="font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; font-size: 11pt; background-color: transparent; font-size: 11pt; background-color: transparent;"></span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">Use of quotation marks varies depending on the shell. The only way R modifies the execution of the configure command is to define the variables listed at the end of this post. The Guide to Writing R Extensions, however, recommends that authors of configure scripts use R to set defaults using R’s config command. Try</span><br /><span style="font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; font-size: 11pt; background-color: transparent; font-size: 11pt; background-color: transparent;"></span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R CMD config --help</span><br /><span style="font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; font-size: 11pt; background-color: transparent; font-size: 11pt; background-color: transparent;"></span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">to see a list of variables R remembers from when it was configured and compiled.</span><br /><span style="font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; font-size: 11pt; background-color: transparent; font-size: 11pt; background-color: transparent;"></span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">When R calls make, it tacks a few files together. The first is the Makevars that configure just customized. The next is a list of variables, mostly from when R, itself, was configured. The last, shlib, is the target to build a shared library.</span><br /><span style="font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; font-size: 11pt; background-color: transparent; font-size: 11pt; background-color: transparent;"></span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">make -f Makevars -f /opt/local/lib/R/etc/x86_64/Makeconf -f /opt/local/lib/R/share/make/shlib.mk SHLIB='foo.so' OBJECTS='foo.o'</span><br /><span style="font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; font-size: 11pt; background-color: transparent; font-size: 11pt; background-color: transparent;"></span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">The only variables not defined explicitly with Makeconf are</span><br /><span style="font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; font-size: 11pt; background-color: transparent; font-size: 11pt; background-color: transparent;"></span><ul style="font-family: Times; font-size: medium; "><li style="list-style-type: disc; font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; "><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">PKG_CFLAGS - Where includes go.</span></li><li style="list-style-type: disc; font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; "><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">PKG_CPPFLAGS - For the C preprocessor, if relevant.</span></li><li style="list-style-type: disc; font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; "><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">PKG_CXXFLAGS - For the C++ compiler.</span></li><li style="list-style-type: disc; font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; "><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">PKG_OBJCFLAGS - Objective C’s CFLAGS.</span></li><li style="list-style-type: disc; font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; "><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">PKG_OBJCXXFLAGS - Objective C++’s CFLAGS.</span></li><li style="list-style-type: disc; font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; "><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">PKG_LIBS - Where we put libraries and the directories that hold them.</span></li></ul><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">These are the only variables we should bother to define within Makevars.in. Everything else, from CXX to CFLAGS, is already explicitly within Makeconf, so tough cookies if we want to change it, unless we are willing to make a custom target in our own Makefile in the src directory.</span><br /><span style="font-family: Arial; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; font-size: 11pt; background-color: transparent; font-size: 11pt; background-color: transparent;"></span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">For our package, foo, we want to give the person installing the software a way to customize the include directories and library locations, so we probably want to check in our configure.ac</span><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; "> for the existence of FOO_CFLAGS and FOO_LIBS and assign those values to PKG_CFLAGS and PKG_LIBS. Using package-specific naming helps when there are multiple packages installed, an using variables at all helps people installing avoid figuring out how to pass command-line arguments to R.</span><br /><h3 style="font-family: Times; font-size: medium; "><span style="font-size: 14pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: bold; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">Sample Variables Defined Before Configure and Make</span></h3><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">AWK=awk</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">DYLD_LIBRARY_PATH=/opt/local/lib/R/lib/x86_64</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">EGREP=/usr/bin/grep -E</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">LN_S=ln -s</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">MAKE=make</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">PAGER=/usr/bin/less</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">PERL=/opt/local/bin/perl</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_ARCH=/x86_64</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_BROWSER=/opt/local/bin/kfmclient</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_BZIPCMD=/opt/local/bin/bzip2</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_DOC_DIR=/opt/local/lib/R/doc</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_GZIPCMD=/opt/local/bin/gzip</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_HOME=/opt/local/lib/R</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_INCLUDE_DIR=/opt/local/lib/R/include</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_LIBRARY_DIR=/Users/username/Documents/rlib/foo.Rcheck</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_LIBS=/Users/ajd27/Documents/rlib/foo.Rcheck</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_LIBS_SITE=</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_LIBS_USER=~/R/x86_64-apple-darwin10.6.0-library/2.12</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_PACKAGE_DIR=/Users/ajd27/Documents/rlib/foo.Rcheck/foo</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_PACKAGE_NAME=foo</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_PAPERSIZE=letter</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_PDFVIEWER=/opt/local/bin/ggv</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_PLATFORM=x86_64-apple-darwin10.6.0</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_PRINTCMD=lpr</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_RD4DVI=ae</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_RD4PDF=times,hyper</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_SESSION_TMPDIR=/var/folders/4Z/4ZJIk-FGFl4hDOSWoQMBmU+++TI/-Tmp-//Rtmp3r18dL</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_SHARE_DIR=/opt/local/lib/R/share</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_TEXI2DVICMD=/opt/local/bin/texi2dvi</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_UNZIPCMD=/usr/bin/unzip</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">R_ZIPCMD=/usr/bin/zip</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">SED=/usr/bin/sed</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">TAR=/usr/bin/gnutar</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">TR=/usr/bin/tr</span><br /><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">WHICH=/usr/bin/which</span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; background-color: transparent; font-family: Times; font-size: medium; "><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; "><br /></span></div>Drew Dolgerthttp://www.blogger.com/profile/10755676286500857195noreply@blogger.com0tag:blogger.com,1999:blog-24489906.post-72678738652507763852010-12-04T23:49:00.004-05:002010-12-05T00:09:52.387-05:00Installing Rmpi under Linux on Ranger at TACCRmpi is a package for the R language to allow it to use MPI to run in parallel. Installation on Ranger is complicated by Ranger's selection of compilers, MPI libraries, and default installed packages. I got it to work, so here are hints.<div><br /></div><div>R's default compiler options for installed packages are the same as those used to compile R. The Ranger copy of R was compiled with gcc. MPI libraries on Ranger are compiled with either PGI or Intel compilers, and there can be incompatibilities loading shared libraries from PGI and Intel into gcc code (especially the version of PGI on Ranger). An easy way to get started healthily is to recompile R with the Intel compilers, so "module swap pgi intel" and configure a home directory installation of R using mpicc, mpicxx, and mpif90 with the Known Good Intel options for Ranger, "-O2 -xW -fPIC".</div><div><br /></div><div>The Rmpi library depends on the R wrapper for the SPRNG library which, of course, depends on SPRNG itself. It also needs gmp. Ranger's gmp library is just fine, so "module add gmp". If you want to know includes and libs for any Ranger library, use "module help <libname>". The SPRNG library has a problem, though, because it wasn't compiled with -fPIC, which is necessary to build shared libraries on this platform.</div><div><br /></div><div>Rmpi needs SPRNG 2.0, no other version. The Ranger sprng2.0 library is compiled without -fPIC, so we make our own. Download it with "curl http://sprng.cs.fsu.edu/Version2.0/sprng2.0b.tar.gz|tar zxf -". Build the MPI version with the Intel compilers, again, using the Known Good Intel Options, "-O2 -xW -fPIC" and mpicc, mpicxx, mpif90, but add another option Fortran flags to help Fortran compatibility, "-assume 2underscores". This comes from a little bug in the SPRNG autoconf, but it only affects Fortran builds. You have to set this information in sprng2.0/make.CHOICES and sprng2.0/src/make.INTEL.</div><div><br /></div><div>SPRNG doesn't make a shared executable. No worries. We can rebuild it as a shared library. Go to the directory of libsprng.a, and repack it with a reference to the gmp library.</div><div><br /></div><div><div>ar -x libsprng.a</div><div>icc -shared -fPIC -Wl,-rpath,$TACC_GMP_LIB -Wl,-zmuldefs *.o -L${TACC_GMP_LIB} -lgmp -o libsprng.so</div></div><div><br /></div><div>Were you to forget to add the reference to -lgmp, you would later see that libsprng.so cannot be loaded by R because it cannot find a function __gmp_cmp.</div><div><br /></div><div>To build rsprng, first download the tar file. Then invoke R's installer with hints about the location of sprng. Mine looked as follows.</div><div><br /></div><div>~/R/bin/R CMD INSTALL --configure-vars='CFLAGS="-I/opt/apps/gmp/4.2.4/include" LDFLAGS="-L/opt/apps/gmp/4.2.4/lib"' --configure-args='--with-sprng=/share/home/00933/tg459569/sprng' rsprng_1.0.tar.gz</div><div><br /></div><div>If you get the paths correct, that should build. Now the Rmpi library wants the same attention as Rsprng. First download it. Then let R build it.</div><div><br /></div><div>~/R/bin/R CMD INSTALL --configure-vars='CFLAGS="-O2 -xW -fPIC" LDFLAGS="-O2 -xW -fPIC"' --configure-args='--with-Rmpi-type=MPICH' Rmpi_0.5-9.tar.gz</div><div><br /></div><div>When R installs, it will fail its first test because miprun won't work on a Ranger login node. Just go to <a href="http://math.acadiau.ca/ACMMaC/Rmpi/index.html">http://math.acadiau.ca/ACMMaC/Rmpi/index.html</a> and try the tutorial in a batch job.</div><div><br /></div><div>HTH</div><div>Drew</div><div><br /></div>Drew Dolgerthttp://www.blogger.com/profile/10755676286500857195noreply@blogger.com2tag:blogger.com,1999:blog-24489906.post-12744366063738077492010-09-03T17:41:00.003-04:002010-09-03T17:50:19.459-04:00A Standard Way to Use Python's Logging ModuleThe documentation for Python's logging module shows you its great features, but it doesn't mention standard usage if you are writing a little application or a library.<br /><br />Make a separate logger for each file, and name the loggers to match the hierarchy of modules. For the file acert/hfesr/Runner.py, it starts with:<br /><br /><pre>import logging<br />logger = logging.getLogger('acert.hfesr.Runner')<br />class Runner(object):<br /> def __init__(self):<br /> logger.debug('Initializing Runner')</pre><br />That's quite simple. Then you can enable and disable logging by file or by module.<br /><br />The second tip is that, if you have written a library, don't use logging.basicConfig() in that library because it makes logging handlers that are difficult for subsequent client applications to quiet.<br /><br />HTHDrew Dolgerthttp://www.blogger.com/profile/10755676286500857195noreply@blogger.com0tag:blogger.com,1999:blog-24489906.post-87221304128292602822010-09-03T17:28:00.002-04:002010-09-03T17:40:49.706-04:00/dev/null for C++ ostreamI often make C++ classes that write to some stream given to them:<br /><br /><span style="font-size:85%;"><span style="font-family:courier new;">Bear::Save(std::ostream& out) {</span><br /><span style="font-family:courier new;"> out << "Fur is " << color << std::endl;</span><br /><span style="font-family:courier new;"> out << "Age is " << age << std::endl;</span><br /><span style="font-family:courier new;">}</span></span><br /><br />Editing somebody's code today, I needed an ostream equivalent of /dev/null, some stream into which a class could write without printing anything. This can be done by creating a stream buffer that never prints.<br /><br /><span style="font-size:85%;"><span style="font-family:courier new;">class nullbuf : public std::basic_streambuf<char,></char,></span><br /><span style="font-family:courier new;">{</span><br /><span style="font-family:courier new;">protected:</span><br /><span style="font-family:courier new;"> virtual int overflow(int c) { return c; }</span><br /><span style="font-family:courier new;">};</span><br /><br /><span style="font-family:courier new;">class nullstream : public std::basic_ostream<char,> > {</char,></span><br /><span style="font-family:courier new;"> nullbuf _streambuf;</span><br /><span style="font-family:courier new;">public:</span><br /><span style="font-family:courier new;"> nullstream() :</span><br /><span style="font-family:courier new;"> std::basic_ostream<char,> &gt(&_streambuf)</char,></span><br /><span style="font-family:courier new;"> { clear(); }</span><br /><br /><span style="font-family:courier new;">};</span><br /></span><br />Using it looks like any other stream:<br /><span style="font-size:85%;"><br /><span style="font-family:courier new;">nullstream dev_null;</span><br /><span style="font-family:courier new;">dev_null << "Nobody will ever see this.";</span><br /><span style="font-family:courier new;">ostream* output = new nullstream();</span><br /><span style="font-family:courier new;">(*output) << "This will be invisible.";</span><br /></span><br />As usual, hope this helps.Drew Dolgerthttp://www.blogger.com/profile/10755676286500857195noreply@blogger.com0tag:blogger.com,1999:blog-24489906.post-78928375637014044082007-05-10T12:53:00.000-04:002007-05-10T13:26:11.073-04:00Converting OpenDX VRML to Texture-mapped VRMLOpenDX typically writes VRML files with vertex colors, so that every vertex has an associated RGB value. Many modeling programs cannot read VRML with vertex colors, so it is useful to convert them. If you are converting a single object or a relatively simple object, there are applications to do the conversion, such as <a href="http://www.unwrap3d.com/">Ultimate Unwrap3D</a>. This program examines the given Shape and figures out how to dice it evenly onto a two-dimensional texture map.<br /><br />VRML from OpenDX is typically special, though, in that all of the color comes from a colormap applied to a scalar value. That means you probably started with every vertex being associated with a number between some min and max and assigned RGB values to the range of numbers. Because of this, we can use a one-dimensional texture to color a VRML from OpenDX. The process is a lot faster than unwrapping the texture, and it doesn't leave any seams.<br /><br />Here's how we did it.<br /><ol><li>Save the VRML as grayscale.</li><li>Save the colormap as a jpeg or tiff (depending on what colormaps your VRML viewer can read).</li><li>Use a custom <a href="http://www.tc.cornell.edu/%7Eajd27/texturize.py.txt">python script</a> to translate each grayscale value into a u-v index into the saved colormap.</li></ol>Because I didn't want to write code to read VRML, I used a shortcut. There is a program called X3D-Edit which translates VRML to X3D, which is essentially VRML in XML form. Then I used Python's ElementTree to process the XML and write a new X3D file. Then X3D-Edit or the standard XSLT will translate X3D back to VRML. It sounds complicated, but the translation from VRML to X3D and back is quite sure, and X3D is a lot easier to parse than VRML.Drew Dolgerthttp://www.blogger.com/profile/10755676286500857195noreply@blogger.com0tag:blogger.com,1999:blog-24489906.post-90647818146367480632007-05-10T12:46:00.000-04:002007-05-10T13:27:53.749-04:00Powershell XML Handling for Empty nodesWindows Powershell handles xml objects a little inconsistently. If a node is empty, then Powershell treats it as a string. If the node is non-empty, then it treats the node as an XmlNode. You can circumvent this behavior by using the Item() interface.<br /><br />PS>$x=[xml]"<root/>"<br />PS>$x.root.GetType()<br /> <p class="MsoNormal">IsPublic IsSerial Name<span style=""> </span>BaseType<o:p></o:p><br />-------- -------- ----<span style=""> </span>--------<o:p></o:p><br />True<span style=""> </span>True<span style=""> </span>String<span style=""> </span><span style=""> </span>System.Object<o:p></o:p></p><p class="MsoNormal">PS>$x.Item("root").GetType()</p><p class="MsoNormal">IsPublic IsSerial Name<span style=""> </span>BaseType<o:p></o:p><br />-------- -------- ----<span style=""> </span>--------<o:p></o:p><br />True<span style=""> </span>False<span style=""> </span>XmlElement<span style=""> </span>System.Xml.XmlLinkedNode<o:p></o:p></p>Drew Dolgerthttp://www.blogger.com/profile/10755676286500857195noreply@blogger.com2tag:blogger.com,1999:blog-24489906.post-31113125780024549522007-04-25T14:19:00.000-04:002007-05-10T13:28:55.331-04:00Vista Update KB930178 FailsAn update failed on my new Vista installation with error 0xc015001a. "c015" seems to refer to Component Based Servicing (CBS), which is the tool that applied the update. 001a generally means it couldn't create or delete something. C:\Windows\WindowsUpdate.log shows more detail about the error, including the directory where it unpacked the distribution. Running Process Monitor from Sysinternals shows that TrustedInstaller.exe failed to create a file in the unpacking directory, C:\Windows\SoftwareDistribution\Download\<long>\inst.<br /><br />I tried turning off Symantec Antivirus. I tried turning off User Access Control (UAC) in the Users and Groups control panel.<br /><br />When it finally worked, the system state was the following:<br />- Logged on as THE administrator.<br />- UAC was off<br />- Changed properties on the SoftwareDistribution\Download\ directory so that it, and its children, were no longer read only.<br />- applied the patch during shutdown<br /><br />Drew</long>Drew Dolgerthttp://www.blogger.com/profile/10755676286500857195noreply@blogger.com1tag:blogger.com,1999:blog-24489906.post-1160409034068242692006-10-09T11:36:00.000-04:002007-05-10T13:29:25.541-04:00Fortran Memory ErrorMy buddy Steve just tracked down a Fortran memory error on Windows. I like recording these debugging sessions and thought to share one.<br /><br />This Fortran90 application couples fluid dynamics and chemistry to do combustion. It uses the <a href="http://www.intel.com/cd/software/products/asmo-na/eng/perflib/mkl/">MKL </a>version of <a href="http://www.netlib.org/lapack/">LAPACK</a>. It failed with a page fault, which means that there was a memory problem. How do you find it?<br /><br />Fortran90 is unlikely to cause memory errors of its own accord. The language doesn't use C-style pointers, although it does add allocatable arrays to the Fortran77 standard. These arrays shouldn't be capable of corruption under normal circumstances.<br /><br />The first step for Steve was to use the PageHeap tool. The old-fashioned way to use this is to use gflags.exe to turn on the PageHeap switch in the operating system. I read there is a fancier way to use it in Visual Studio 2005, but gflags.exe will suffice.<br /><br />Turning on PageHeap made the program crash much sooner than before. It crashed in a particular function during exit from that function. Pseudocode for the function follows.<br /><br /><span style="font-family:lucida grande;"><span style="font-family:arial;">Sub DoWork(array,size)</span><br /><span style="font-family:arial;"> If size>N</span><br /><span style="font-family:arial;"> array = Allocate(4000)</span><br /><span style="font-family:arial;"> end if</span><br /><span style="font-family:arial;">End Sub</span><br /><br /><span style="font-family:georgia;">This function only failed when allocate was </span><span style="font-style: italic;font-family:georgia;" >not</span><span style="font-family:georgia;"> called. We knew that this subroutine implicitly calls deallocate when it exits, so it seemed like there might be an invalid array pointer in the heap. That means that the heap corruption discovered during the implicit deallocation was caused by stack corruption which claimed that the array had been allocated.</span><br /><br /><span style="font-family:georgia;">Steve's friend had the bright idea to use this function as a test to find where the corruption first occurred. He moved it higher and higher in the code until he found a function call to LAPACK. Looking closely at that function call, he realized that he had called LAPACK to process a matrix but had given it the wrong size for the matrix. LAPACK was overwriting past the matrix boundary. Why was it able to do this? The LAPACK libraries are Fortran77, therefore without a header file. Fortran90 was unable to check that the arguments were correct.<br /><br />- Drew<br /><br /></span></span>Drew Dolgerthttp://www.blogger.com/profile/10755676286500857195noreply@blogger.com0tag:blogger.com,1999:blog-24489906.post-1152810547259026252006-07-13T13:04:00.000-04:002007-05-10T13:29:46.088-04:00Finally Fixed VTK .NET Widget BugI thought there was a subtle interaction between .NET and VTK that caused threading problems during user interaction. The result was that widgets did not always work. Picking would fail. The problem turned out to be that I had defined an enum for EventIds that was a copy of the VTK enum. VTK, itself, changed these values from version 5.0 to the development version, 5.x, and I was using the 5.x enum for the 5.0 code.<br /><br />The fix for what I thought was a subtle bug is just to use integers instead of the EventId enum. For instance, replace EventIds.EndPickEvent with the number 9. You can look up these values in VTK\Common\vtkCommand.h.<br /><br />There are no other real bugs in VTK .NET, that I know of. I could do a few things to make debugging easier and wrap HWND values, but that's about it.<br /><br />That means I can release a version of VTK .NET for VTK 5.0.1. I've already integrated the new release into the patch code, tested the code for the new version, and will package it as soon as I can.Drew Dolgerthttp://www.blogger.com/profile/10755676286500857195noreply@blogger.com3tag:blogger.com,1999:blog-24489906.post-1149480095366741062006-06-04T23:50:00.000-04:002006-06-05T00:01:35.376-04:00Compilation of Debugging Tools<span style="font-style: italic;">Debugging Applications for Microsoft .NET and Microsoft Windows</span> by Robbins has been a great, great help to me. Tonight, I used Visual Studio 2005 to compile several of the utilities supplied with the book and had to make a few changes to upgrade from Visual Studio .NET.<br /><br />For CrashFinder, change the return value of OnNcHitTest to LRESULT in About.{h,cpp} and Statlink.{h,cpp}. The tricky bit is to add #define _USE_32BIT_TIME_T to stdafx.h. Then you can either modify all of the string handling routines or unselect Treat Warnings as Errors to complete the compilation.<br /><br />Robbins recommends creating map files with the linker options /MAPINFO:EXPORTS and /MAPINFO:LINES. The LINES option in Visual Studio 2005's linker has been removed, but you can resurrect that information using pdb2map from the book's CD. That's the only way I know to get this information.Drew Dolgerthttp://www.blogger.com/profile/10755676286500857195noreply@blogger.com0tag:blogger.com,1999:blog-24489906.post-1147359747057225582006-05-11T10:49:00.001-04:002006-06-01T11:44:23.166-04:00VTK .NET to Sourceforge<p>I posted VTK .NET on 23 March and have had about one hundred people come to the site each week. I can see that people download examples, so it seems to run well. I have submitted a request for a <a href="http://www.sourceforge.net">Sourceforge</a> site, using the BSD license, which is the same that VTK uses. That should take another week or two to process.</p><br /><h3>Bugs Discovered</h3><br /><ul><li>I did not modify CMakeLists.txt for vtkMy or vtkLocal, which are the standard ways to wrap your own VTK C++ code for wrapper languages.</li><br /><li>There seems to be a problem with vtk3DWidgets. They mostly work, but a few things do not work. For example, annotatePick.cs from the examples does not notify you of new picks.</li></ul><br /><h3>What People Want Next</h3><br /><ul><li>I saw a comment that the previous version of VTK .NET needed to wrap HWND types so that you could build a Windows Forms Control in C#. I've built one in managed C++, but I see the logic in the request, and I think I can add that translation to the wrapper code.</li><br /><li>Two people who emailed me have commented that they are working on using VTK .NET within ASP.NET. One wanted to use it to generate images on the server side. The other is interested in the more ambitious project of providing manipulable 3D graphics, something that interests me, too. There are a number of challenges to making this happen, but it would clearly be a great thing to implement.</li></ul>Drew Dolgerthttp://www.blogger.com/profile/10755676286500857195noreply@blogger.com5tag:blogger.com,1999:blog-24489906.post-1146247220270567142006-04-28T13:55:00.000-04:002006-04-28T15:28:14.183-04:00SciPy distribution for P3 fails - what illegal instruction meansI have been trying to help someone install <a href="http://www.scipy.org/">SciPy</a> on our clusters. The site provides a package for P4 which works fine and one for PIII which works on my laptop but fails on our test login node. I downloaded <a href="http://www.cpuid.com/cpuz.php">CPU-Z</a> to check things out, and it turns out that my Pentium M (Dothan) laptop supports MMX, SSE, and SSE2 while the dual Pentium III-S ancient server for testing supports only MMX and SSE.<br /><br />As a result the provided installer for Python 2.4 and Pentium III fails for older P3 machines.<br /><pre class="code">scipy-0.4.8.win32-py2.4-pentium3.exe</pre><br />The test code from that provided distribution, scipy.test() will fail on older P3 chips (such as PIII-S, Tualatin) in several of the package tests with a Dr. Watson error of<br /><pre class="code">Unhandled exception at 0x6988d3f7 in python.exe:<br />0xC000001D: Illegal instruction</pre><br />The supplied binaries for ATLAS on a Windows P3 box do not have this problem. I built a running package using online instructions for a MinGW build using the precompiled ATLAS binaries. It is a quick process because you don't have to rebuild ATLAS.<br /><br />I'm pretty sure we didn't just accidentally install the P4 version because I checked the show_config at the time, and we did it twice. I can't prove it because I didn't save the output. Also, the P4 version fails quickly on the P3, but the fewer tests fail when using an SSE2 P3 version on an older P3.<br /><br />On the PIII-S where the install failed, the CPU information is<br />has_mmx has_sse is_32bit is_Intel is_PentiumIII is686<br />The ATLAS archdef for both the supplied package and my working build was PIII/gcc/misc and mmdef was PIII/gcc/gemm.<br /><br />We run on Python 2.4.2, Windows XP Service Pack 2<br /><br />The instructions that failed included cvtsi2sd, movsd, and ucomisd. I found them listed in the MMX section of the Intel assembly specification, but they are included listed only for SSE2 support. For a sense of what the code looks like, the following shows the assembly snippet.<br /><pre class="code">684C5825 mov dword ptr [esp+68h],0 <br />684C582D mov eax,dword ptr [edx] <br />684C582F mov edx,dword ptr [esi] <br />684C5831 cmp eax,edx <br />684C5833 cmovg eax,edx <br />684C5836 cvtsi2sd xmm0,eax <br />684C583A movsd mmword ptr [esp+14h],xmm0 <br />684C5840 fld qword ptr [esp+14h] <br />684C5844 mov dword ptr [esp+5Ch],0 <br />684C584C fmul qword ptr ds:[684C5538h] <br /><br />EAX = 0000000F EBX = 684C550E ECX = 00902DD0 EDX = 0000000F <br />ESI = 0021EB6C EDI = 0021EB60 EIP = 684C5836 ESP = 0021E8A4 <br />EBP = 00000001 EFL = 00010216</pre><br />The instruction pointer is at the cvtsi2sd instruction. Normally, exceptions are thrown by the instruction before the instruction pointer, but here the exception happened just before the instruction pointer would be incremented.<br /><br />Drew DolgertDrew Dolgerthttp://www.blogger.com/profile/10755676286500857195noreply@blogger.com0tag:blogger.com,1999:blog-24489906.post-1144705514371122112006-04-10T17:43:00.000-04:002006-04-10T17:45:14.393-04:00Debugging Mixed Interop Native and Managed Assemblies with Visual Studio 2005<p>I've been trying to track down a bug in a managed C++ wrapper around native C++ classes, and Visual Studio 2005 won't allow me to step directly from the managed code into the native code. I know it is possible, but this is the only way I see to do it:</p><br /><ol><br /><li>Put a System.Diagnostics.Debug.Assert(false) near the start of the application.</li><br /><li>Run the application and accept the option to debug it.</li><br /><li>Opt to "manually choose debugger," and choose both native and managed.</li><br /><li>Visual Studio 2005 then allows me to step into the native code.</li><br /></ol><br /><p>Drew </p>Drew Dolgerthttp://www.blogger.com/profile/10755676286500857195noreply@blogger.com0tag:blogger.com,1999:blog-24489906.post-1143127417281062622006-03-23T10:15:00.000-05:002006-04-10T10:31:34.393-04:00unmanaged Visual C++ for Visual Studio 2005 causes (soluble) dll loading problemsHi All,<br /><br />This message is about where dlls go when they are part of “isolated assemblies.” Everyday C++ users may run into this when they move their code from a development machine to a node on a compute cluster.<br /><br />1. When you copy an executable to the cluster, it now needs not only dlls but also manifest files copied with it, or we have to install them on the cluster for the users.<br /><br />2. There is a common occurrence where a failure in the linker can cause an executable to fail ever to load. There is a quick fix for this.<br /><br /><strong>Part I:<br /></strong>When compiling unmanaged visual C++ applications in VS2005, you have to pay attention to how you distribute the dlls. Normally, you just put Dlls beside the exe. That isn’t working for some good reasons. I’m going to give you an outline here of where the bodies lie, trying to show you pertinent directories, tools, and project settings.<br /><br />Google keywords: Isolated applications, side-by-side assemblies, winsxs<br />Web pages:<br />About isolated apps and assemblies<br /><a title="http://msdn2.microsoft.com/en-us/library/ms235342(en-US,VS.80).aspx" href="http://msdn2.microsoft.com/en-us/library/ms235342(en-US,VS.80).aspx">http://msdn2.microsoft.com/en-us/library/ms235342(en-US,VS.80).aspx</a><br />blog with details on how to do installation with isolated apps<br /><a title="http://blogs.msdn.com/nikolad/" href="http://blogs.msdn.com/nikolad/">http://blogs.msdn.com/nikolad/</a><br />Windows Installer XML – use xml file to create an msi.<br /><a title="http://sourceforge.net/projects/wix" href="http://sourceforge.net/projects/wix">http://sourceforge.net/projects/wix</a><br /><br />There is a section in VS under Linker->Manifests that says you can<br />- embed manifest – yes/no<br />- allow isolation – yes/no<br /><br />Visual Studio now distributes the CRT and debug CRT and versions of ATL and MFC libraries in VS<br /><span style="font-family:courier new;font-size:85%;color:#3333ff;">C:\Program Files\Microsoft Visual Studio 8\VC\redist\x86<br />C:\Program Files\Microsoft Visual Studio 8\VC\redist\Debug_NonRedist\x86<br /></span><br />Here you find the directory Microsoft.VC80.DebugCRT\ with the files<br /><span style="font-family:courier new;font-size:85%;color:#3333ff;">- Microsoft.VC80.DebugCRT.manifest<br />- msvcm80d.dll<br />- msvcp80d.dll<br />- msvcr80d.dll<br /></span><br />Executables, under various conditions, are supposed to be able to find dlls in various directories if you put the manifest with them<br /><span style="font-family:courier new;font-size:85%;color:#3333ff;">- C:\Windows\WinSXS<br />- <appdirectory><br />- <appdirectory>\<appname>.exe.local<br />- <appdirectory>\<manifestname></span><br /><br /><strong>Part II: The Common Deal-breaker: Oldnames.lib<br /></strong><br />When you compile code which links to libraries using older, deprecated methods, the linker finds those methods in an implicit library called Oldnames.lib. Because of a flaw in the linker, any time it pulls in oldnames.lib, it will also pull in a copy of the wrong CRT. That is, if you link against the debug CRT, it will pull in the release CRT, and vice versa. This wouldn’t normally cause the executable to fail to run, except that the linker includes only the correct assembly. For instance, when linking against the debug CRT, the following manifest is included in the executable, even though the linker mistakenly linked to the runtime CRT, as well.<br /><br /><span style="font-family:courier new;font-size:85%;color:#3333ff;"><br /><assembly xmlns="'urn:schemas-microsoft-com:asm.v1'" manifestversion="'1.0'"><br /> <dependency><br /> <dependentassembly><br /> <assemblyidentity type="'win32'" name="'Microsoft.VC80.DebugCRT'" version="'8.0.50608.0'" processorarchitecture="'x86'" publickeytoken="'1fc8b3b9a1e18e3b'"><br /> </dependentassembly><br /> </dependency><br /></assembly><br /></span><br />The solution is to tell the linker to ignore the other CRT. In other words, if you link with msvcrtd.lib, you have to explicitly tell the linker to ignore msvcrt.lib, and vice versa. That makes the linker think longer about where it can find the references it needs, and it comes back around to the correct library.<br /><br />Mind that Microsoft would not consider this problem a linker flaw because I have only seen it happen when linking to libraries older than my current version of Visual Studio, and linking Visual Studio 2005 projects to those of Visual Studio 2003 or Visual Studio 6 is <em>not supported.</em> This is likely confounded with the fact that I am linking release and debug always to a release library, which is <em>not supported.</em> Does any company really distribute their development libraries as MT, MD, MTd and MDd? I've only seen VRCO do this.<br /><br />At this point, I’ve also created an installer for the C runtimes, as described in the blog link above, so that’s what is loading. I just run vccrt.msi, then my application runs. I used this tool called <a href="http://sourceforge.net/projects/wix">WIX</a> to install it. It’s a bit of open source from Microsoft. The project doesn't appear to be under development any longer, but it's still working for now.<br /><br />DrewDrew Dolgerthttp://www.blogger.com/profile/10755676286500857195noreply@blogger.com0tag:blogger.com,1999:blog-24489906.post-1143126715472234952006-03-23T10:09:00.000-05:002006-03-23T11:02:27.533-05:00running openmp on cluster requires per-application configuration fileJohn, Dave, and I were trying to run a Visual Studio 2005 OpenMP code on cluster nodes. We ran into an extra wrinkle with the isolated assembly for Visual Studio’s OpenMP. In addition, a curiously low-tech bug in Visual Studio kept us from discovering we had the correct solution.<br /><br /><strong>Typical deployment of isolated assemblies on a cluster node</strong><br />We have already covered the typical story for isolated assemblies. If an application uses a dll which was compiled as an isolated assembly, then the application has to contain a manifest which specifies those assemblies. Visual Studio automatically embeds that manifest in the application. When you deploy that application to a new machine, Microsoft recommends that you use an installer to install that assembly as a shared assembly, meaning that the installer places the assembly in the WinSxS folder in the Windows folder, which functions like a GAC. Visual Studio even includes a vcredist.exe program to install all of the release versions of Visual Studio isolated assemblies into the WinSxS folder.<br /><br />Because we are running on cluster nodes, we want to know how to deploy those isolated assemblies, instead, as private assemblies. This is usually a simple matter. You find directory for the isolated assembly, such as Microsoft.VC80.DebugCRT, in C:\Program Files\Visual Studio 8\VC\redist\Debug_NonRedist\x86. Copy the whole directory to the same directory as your executable on the cluster node, and everything should work.<br /><br />By the way, Dave installed the redistributable, release version, isolated assemblies on one of the clusters. We have to figure out whether we will install all Visual Studio assemblies on the clusters for the users. The problem will still exist for other isolated assemblies, although we don’t, at this point, know of any coming from companies other than Microsoft or products other than Visual Studio.<br /><br /><strong>How OpenMP requires another step to deploy</strong><br/><br />When you try the above technique for OpenMP, the executable still fails to find the assembly on the cluster node. When Visual Studio compiles your OpenMP program, it embeds a manifest file that refers to version 8.0.50608.0 of the OpenMP assembly, but the actual assembly is version is 8.0.50727.42. Why would this work on your machine? It works because your computer has a machine-level policy, sitting in %windows%\WinSxS\Policies, which states that, if any application asks for version 8.0.50608.0, it’s okay to give them 8.0.50727.42.<br /><br />The version mismatch between the distributed assembly and the assembly required by the compiler is a <em>known problem</em>. It happens because different parts of Visual Studio are frozen at differnt times before delivery. What we have to do to make this run on the cluster is to deploy that same policy as an application configuration file.<br /><br />Application configuration files for native application files are very similar to those for managed files. If your program is myprog.exe, you make a text file, with UTF-8 encoding (with or without a byte-order mark), called myprog.exe.config. In it, you redirect the program’s binding to the newer version of the debug OpenMP assembly. Note that, when you identify the DebugOpenMP assembly, you leave out the version attribute because you will specify it in the bindingRedirect node.<br /><br /><span style="color:#3333ff;"><?xml version="1.0" encoding="UTF-8" standalone="yes"?><br /><configuration><br /> <windows><br /> <assemblybinding xmlns="urn:schemas-microsoft-com:asm.v1"><br /> <assemblyidentity name="myprog.exe" processorarchitecture="x86" version="1.0.0.1" type="win32"><br /> <dependentassembly><br /> <assemblyIdentity type="win32" name="Microsoft.VC80.DebugOpenMP"<br /> processorArchitecture="x86"<br /> publicKeyToken="1fc8b3b9a1e18e3b"/><br /> <bindingredirect oldversion="8.0.50608.0" newversion="8.0.50727.42"><br /> </dependentassembly><br /> </assemblybinding><br /> </windows><br /></configuration><br /></span><br /><br />If we had deployed the OpenMP assembly to WinSxS, so that it was a shared assembly, I think that there would have been a machine-level policy to take care of the version mismatch. As is, we had to make this file for ourselves and deploy it in the same directory as myprog.exe.<br /><br /><strong>How a little bug made this hard to figure out</strong><br />Yesterday morning, we had figured out that there was a version mismatch and that an application configuration file was necessary. We had copied the debug OpenMP assembly directory into the same directory as our application, but nothing worked. After five hours, we noticed that the debug OpenMP assembly directory was called Micrsoroft.VC80.DebugOpenMP. If you look in your Studio 2005 VC\Debug_NonRedist\x86 directory, you’ll find that the directory name is wrong.<br /><br />Drew DolgertDrew Dolgerthttp://www.blogger.com/profile/10755676286500857195noreply@blogger.com1tag:blogger.com,1999:blog-24489906.post-1143125445847759822006-03-23T09:38:00.000-05:002006-06-16T07:31:37.440-04:00Attempting to contribute to VTK open sourceWe use the <a href="http://www.kitware.com/vtk/">Visualization Toolkit</a> a lot around here to make standalone applications or to use in our <a href="http://www.tc.cornell.edu/Services/Visualization/cave.htm">CAVE</a>. I usually write the applications in Python using <a href="http://www.wxpython.org/">wxPython</a>, but sometimes the client needs an installer and a regular windows application, so I like to use Windows Forms in that case. To that end, I normally wrap VTK code in managed C++. There were, and still are, compiled wrappers provided by <a href="http://herakles.zcu.cz/research/vtk.net/">a group in Slovenia</a>, but this seems to have been a masters thesis, and they never released the code.<br /><br />It seemed like a great chance to contribute to the project, so I've <a href="http://www.tc.cornell.edu/~ajd27/VTK/DotNetWrap.html">created wrappers</a> that could fold into the source tree.<br /><ul><li>Managed C++ wrappers for VTK.</li><li>Allow use in C#, managed C++, VB, and others.</li><li>.NET 2.0 only, meaning Visual Studio 2005.</li><li>Builds within the typical CMake build system for VTK.</li><li>Includes a .NET Windows Forms Control for drag-and-drop in Visual Studio designer.</li></ul><p>We'll have to see how successful these are. Because the CMake build system does not handle managed C++ projects, the build isn't entirely hands-free. Just before hitting Build in Visual Studio, the user has to execute a macro to convert the projects to managed code. That problem means this wrapper couldn't be built in nightly dashboard tests. I'd be willing to extend CMake to handle managed projects if that's what it takes.</p><p>Drew Dolgert</p>Drew Dolgerthttp://www.blogger.com/profile/10755676286500857195noreply@blogger.com0tag:blogger.com,1999:blog-24489906.post-1142976354599910352006-03-21T16:25:00.000-05:002006-04-10T10:24:11.963-04:00WelcomeI work on scientific simulations and visualization. Some of my friends plan to chip in. We are driven to start this blog by having spent too much time solving technical problems we are sure others have faced. Welcome.Drew Dolgerthttp://www.blogger.com/profile/10755676286500857195noreply@blogger.com0