This is a README, so READ ME!

#	Starting from scratch?
#	You will need 'git' installed.  If using a Mac, probably best
#	installed via MacPorts' 'port' command, which you may need
#	to install as well.  You may also need to install qt5 or qt5-mac.
#	Installation of these binaries, ruby and rubygems is beyond
#	the scope of this README file.

git clone [email protected]:ccls/odms.git

#	Or just give it a pull to update.
git pull

#	create a big hash (rake secret) and insert as your secret token
cp config/initializers/secret_token.rb.example config/initializers/secret_token.rb

#	define your databases
cp config/database.yml.example config/database.yml

#	you may have to copy in some mail credentials ...
cp config/initializers/mail.rb.example config/initializers/mail.rb

#	you may have to copy some other example files like ...
cp config/secrets.yml.example config/secrets.yml
#	This will also require that you use 'rake secret' to populate it.

#	Assuming that ruby 1.9.3 is installed which now includes rubygems.
#	Should probably update to the latest rubygems (1.8.24 as of 20120619)
#	(Not really sure if this is actually necessary in 1.9.3,
#	 but I did it and everything still seems to work.)
#	Using ruby 2.1 (as of 20141203)
#	Using ruby 2.3 (as of 20150117)
gem update --system

#	Install bundler which will be managing all of the gems
gem install bundler

#	Install all of the gems
bundle install

#	drop existing dev db.
#	This is obviously destructive so don't do it 
#	unless you know that.
mysqladmin -u root drop odms_development

#	create new empty dev db
mysqladmin -u root create odms_development

#	create all the other databases (could skip the above create)
rake db:create:all

#	Run all of the migrations.
rake db:migrate

#	Perhaps prep the test db
rake db:test:prepare

#	The test fixtures contain REAL data to be used in
#	the REAL application and so it MUST be installed.
#	Install most test fixtures in database.
#	This is destructive and will remove all existing
#	before adding those in the fixtures.
#rake app:update

#	OR Install zip_code and county csv fixtures in database (takes a while)
#	( this will do app:update as well )
#	As the zip_code and county info doesn't really ever change,
#	I separated it out.  As Rails 3 doesn't do csv fixtures,
#	I had to custom write the task as well.
#rake app:full_update

#	Add expected users and roles (must be after roles are loaded)
#	This isn't destructive, so you can call this as many times as you like.
#rake app:users:add_default

#	Install "production" pages and guides if necessary (as of Mar 25, 2012)
#	This is also destructive.  Once in production, you may never
#	want to do this again as the pages and guides can be maintained
#	through the application.
#rake db:fixtures:load FIXTURES_PATH=production/fixtures/ FIXTURES=pages,guides 

#	For the moment, we are trying sunspot and solr for subject indexing.
#	All seems to be working well, but sunspot needs to be started
#	before the app. This may require RAILS_ENV=yourenvironment
#	(test,development,production).
#	java -XX:+PrintFlagsFinal -version
bundle exec rake sunspot:solr:start

#	start the server (common development mode)
script/rails server

#
#	That will use the development environment by default unless
#	the environment variable RAILS_ENV is set to something
#	or the option -e or --environment is given.
#
#	In addition, this will default to port 3000 unless
#	the -p or --port option is given.
#
#	This will also run the server in non-Daemon mode
#	unless the -d or --daemon option is given.
#	Daemon mode effectively starts the server in the background.
#
#	This will use Webrick by default.  I don't know if it
#	make much difference, but you can also specify mongrel.
#

#	OR start the server on our production server
#	(RAILS_ENV is set in .tcshrc so not NEEDED.)
bundle exec rake sunspot:solr:start RAILS_ENV=production
script/rails server -p 3000 -e production -d mongrel

#	You should also be able to run
script/rails console

#	As well as ...
autotest

#	Sunspot should be started first
bundle exec rake sunspot:solr:start RAILS_ENV=test

#	One pass, running all tests ...
bundle exec rake test

#	Or, since this is the default task, just ...
bundle exec rake

#	One pass, running all tests and checking code coverage ...
#rake test:coverage
#	DO NOT edit the app while this is running, or the resulting
#	coverage information will be corrupt.
#	Actuallly, don't do this at all as using simplecov now

#	Continual testing triggered by file changes ...
autotest
#	Be aware that changes don't necessarily trigger
#	all necessary testing of the given change.
#	They just trigger the associated mapped files.

As this code is stored in publicly accessible repositories, DO NOT UNDER ANY CIRCUMSTANCES PUT ANY REAL SUBJECT DATA IN THE CODE!

Developers will certainly disagree with what to test and what not to test. I test A LOT. I test too much, perhaps. So much so that changing one thing can cause tests to fail way over there. I would rather that happen than having the app fail. I have tried to focus my tests so things are clearer, but there is certainly much more work to do in this arena. There are over 7000 tests and 27000 assertions as of right now. 23000 lines of code in the app and tests. It has become a daunting task to do much of anything without some repercussions.

This will create links like …

<a href=“/icf_master_tracker_updates/5806” class=“button” data-confirm=“Are you sure?” data-method=“delete” rel=“nofollow”>Destroy</a></td>

which contain a number of extra attributes. These extra attributes are not valid html. They are meant to be used by javascript confirmation, but they are still not valid html. So, since I’m not using them, I’ll replace these links with forms like the others.

railscasts.com/episodes/202-active-record-queries-in-rails-3

railscasts.com/episodes/215-advanced-queries-in-rails-3

railscasts.com/episodes/239-activerecord-relation-walkthrough

This is looking good.

def self.no_change_in_query(*args)
 scoped	#	just return scoped to do nothing apparently
end

20141009 scoped no longer exists. The “none” method does now though. Its a little different.

Repeated joins seem to be ignored. Yay!

> User.joins(:roles).joins(:roles).joins(:roles).joins(:roles).joins(:roles).joins(:roles).to_sql
=> "SELECT `users`.* FROM `users` INNER JOIN `roles_users` 
    ON `roles_users`.`user_id` = `users`.`id` INNER JOIN `roles` 
    ON `roles`.`id` = `roles_users`.`role_id`"

def self.with_role_name(role_name=nil)
 ( role_name.blank? ) ? scoped :
  joins(:roles).where("roles.name".to_sym => role_name)
end

I do not like this as it takes precedence over your search unless you add ‘unscoped’ to your chain. It has caused me much grief. It caused grief with acts_as_list not properly scoping. It caused grief with my ZipCode limit not destroying all when I told it to destroy_all because it had “limit(10)” as part of its default_scope. Conclusion. DO NOT USE DEFAULT_SCOPE.

CANNOT have leading 0’s in a string that is being printed as a decimal as ruby thinks its octal and converts. Unless, of course, converting to octal was the intention.

>> sprintf("%06d","0001234")
=> "000668"

CANNOT have leading 0’s and include an 8 or 9 in a string that is being printed as a decimal as ruby thinks its octal and 8 and 9 are invalid octal characters so convert back to Integer first.

>> sprintf("%06d","0001280")
ArgumentError: invalid value for Integer: "0001280"
from (irb):24:in `sprintf'
from (irb):24

Don’t have validations that could raise errors that the user can’t do anything about.

When using validates_presence_of, use allow_blank => true on other validations to avoid multiple errors if blank.

When validating the presence of associations, validate on foreign key rather than association so that the error shows up correctly in view. Unfortunately, this will not guarantee that an association actually exists with that id as it would if the validation was on the association.

The following will not raise error on project_id so in the view, the project_id selector won’t have an error and so won’t be highlighted like other errors will be.

validates_presence_of :project

The following will not ensure that an actual project for the given id exists, but will show the error correctly in the view.

validates_presence_of :project_id

I would like to have a validation that ensures that the _id is legitimate. I do not think that validates_associated is what I want. Having the both does what I want, but then the view will show “Project is blank” twice.

This seems to work …

validates_presence_of :project, :if => :project_id
validates_presence_of :project_id

… but is a bit of a misnomer. The association validation will raise a ‘blank’ error, rather than a more appropriate ‘non-existant’ error, although the message could be altered.

Still some models and associations may need done due to priorities and the fact that some never end up in a view and the association may be problematic. (Aliquot,HomexOutcome,Patient)

Validating the content type of a paperclip attachment seems to be a bit tricky. A big problem seems to be the possibility of multiple content types. The paperclip validation seems to always fail.

validates_attachment_content_type :csv_file,
  :content_type => ["text/csv","text/plain","application/vnd.ms-excel"]

It seems like it almost expects the attachment to be ALL of the content types?

So, using the standard rails ‘validates inclusion of’ seems to work quite well.

validates_inclusion_of :csv_file_content_type,
  :in => ["text/csv","text/plain","application/vnd.ms-excel"],
  :allow_blank => true

It appears that the “newer” classes know how to compare with the “older” ones, but not vice versa, which makes sense.

My comparisons in Sample should just have the “newer” class, the *_at field, first and compare it to the “older” *_on field. OR convert the DateTime to a date. Comparison of a Date to a DateTime will be confusing as it doesn’t just compare the date, it takes the time zone into account which isn’t 100% obvious. I think that it converts it to UTC before comparing. This could effectively make the comparison incorrect.

YES. Add a “to_date” to the _at fields.

>> Time.now
=> Tue Mar 06 20:00:06 -0800 2012
>> Time.now.class
=> Time

>> Time.zone.now
=> Wed, 07 Mar 2012 03:54:31 UTC +00:00
>> Time.zone.now.class
=> ActiveSupport::TimeWithZone

>> Time.zone.now > Date.today
=> true

>> Date.today < Time.zone.now
ArgumentError: comparison of Date with ActiveSupport::TimeWithZone failed
	from (irb):49:in `<'
	from (irb):49

>> Date.today < Time.now
ArgumentError: comparison of Date with Time failed
	from (irb):50:in `<'
	from (irb):50

Also, even if the *_at field is given a Date value, it will be typecast to ActiveSupport::TimeWithZone so there is no terrible need to update all of the tests.

Unit tests are just simple tests, but are generally used to test the model. Functional tests are paired with a controller based on some simple conventions pulled from the test name. Integration tests are a little funky. They aren’t quite as popular and are very application specific. They aren’t tied to anything and and kinda meant to allow for testing a whole situation. I needed them for testing all of our javascript.

Initially, I tried using WebRat, but that does not test javascript. I then tried Capybara, with Selenium, which did test the javascript, but actually had to open a browser and only worked with Firefox, or Chrome with a special driver added. These were cool, but then I found Capybara-Webkit works as a simulated browser with Capybara. This is now the chosen setup for integration testing,

In these integration tests, the Capybara DSL, which has commands like ‘visit’, ‘find_field’, ‘fill_in’, etc., can be executed on the ‘page’ object, but is not necessary as these methods appear to be directly available to the test.

Be advised that some of the integration tests will intermittantly fail due to their threaded nature. I’ve added several ‘wait_until’ blocks which seems to work just fine most of the time. In addition, I upped the default time out. Now they rarely fail, but when they do, it is usually due to a timeout. This is mostly noticed when clicking a link or button and then expecting a change in the database. Sometimes there is none when there should be and then the assert_difference fails. As this behaviour is intermittant, it is a bit difficult to debug. On top of this, when these failures occur they seem to leave some stuff in the database, or put it in after the ROLLBACK, or something. Regardless, stuff gets left in the database and some completely unrelated tests may fail as this content is unexpected. Other than stopping the tests and running “rake db:test:prepare” to clear out the entire database, I have found no viable solution. I could re-write those tests that react poorly to this content, to be more relative than absolute, but this mayn’t always be possible. I could also possibly figure out how to stop the actual tainting of the database. I just upped the timeout to 6 seconds. In addition to the counts failing, existing data could include data which is required to be unique and may trigger a RecordInvalid as well. The key here is really to ensure that the database doesn’t get tainted.

Also note, that while has_css? can take an options hash with :visible, like … has_css?(‘div.phone_numbers div.historic’, :visible => false) … has_field? DOES NOT. It also does not correctly respond the the visible? method. If needing to test the presence and visibility of a form field, use find_field and then visible?.

ie. use this …

assert !page.find_field(
  "study_subject[subject_languages_attributes][2][other_language]").visible?

and not this …

assert page.has_field?(
  "study_subject[subject_languages_attributes][2][other_language]", 
  :visible => false)

as the latter will be true if the field is there regardless of if it is visible

20141007 Some weird stuff happened on a recent update, so some of this may be invalid now.

I noticed that occassionally something fails to rollback properly leaving some records in the database. This can and does causes some odd and confusing errors. Some of the difference tests are just wrong and it isn’t completely clear why. I believe that the integration tests are the initial cause as they are threaded. If something times out or errors in the integration test, things can get kinda funky. I have considered adding a couple destroy_alls, but that really slows things down, and I think would just undo on the rollback anyway. So this appears to be a framework issue. The only “fix” that I have found is to rerun “rake db:test:prepare” which wipes the test database clean. Just an FYI.

Also, recently rather than getting a clear link to a page showing html validation errors if and when they occur, an error similar to

Encoding::UndefinedConversionError: "\xE2" from ASCII-8BIT to UTF-8

is getting raised. This usually means that something on the resulting pages is invalid html. Good luck with that.

This hasn’t happened in a while.

Another odd error. If using a field named “key”, you cannot use it as a group because it generates “key AS key” sql which apparently is syntactically incorrect.

ie. Project.group(‘key’)

Mysql2::Error: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ‘key’ at line 1: SELECT ‘projects`.* FROM `projects` GROUP BY key ActiveRecord::StatementInvalid: Mysql2::Error: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ’key’ at line 1: SELECT ‘projects`.* FROM `projects` GROUP BY key

ActiveRecord::StatementInvalid: Mysql2::Error: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ‘key AS key FROM `projects` GROUP BY key’ at line 1: SELECT COUNT(*) AS count_all, key AS key FROM ‘projects` GROUP BY key

/config/api.yml ( probably removing )
/config/database.yml
/config/initializers/mail.rb
/config/initializers/secret_token.rb

• port qt5-mac (for capybara-webkit gem)

• gem webkit (not used anymore so should remove from Gemfile) REMOVED!

• gem capybara

• gem selenium-webdriver (not used anymore but seems required by others)

• gem capybara-webkit

I have chosen to manage the input of other users by maintaining different repositories and merging as necessary. Using the same repo and creating branches was and is an option and I reserve the right to go that route if it looks better.

initially …

git remote add magee git@github.com:magee/odms.git

then when needed

git fetch magee
git merge --no-commit --no-ff magee/master

As I am the only developer now, this is not used, but could be.

I installed the mysql2 gem and ran … to test the overall speed difference.

time rake odms_import:csv_data

Using just mysql

=> 4061.933u 70.565s 1:21:40.63 84.3%	0+0k 890+638io 0pf+0w

Using mysql2

=> 1878.205u 48.815s 44:48.35 71.6%	0+0k 851+497io 0pf+0w

Not surprisingly, mysql2 seems to be somewhere near twice as fast.

This seems to be working fine in development, testing and production. It is not necessary, but seems to be faster. If desired, all that would need to be done to downgrade to mysql is changing the database.yml adapter values.

I just stumbled upon some issues when using fields_for. When using fields_for on an existing object, it will add a hidden id field. If called on the same existing object multiple times, it will add multiple id fields which will have the same HTML id field which is invalid HTML. It will still work, but my integration tests just failed as the generated HTML document is just empty. The empty document means that there was never a button or link to click. No fields to fill in or select. Nothing. I found this as I was using a more complete object with existing relationship objects in the integration tests, but NOT in the functional tests. The functional tests would have pointed out the validation error.

So. Be very thorough in the functional tests.

Only call fields_for on an object ONCE PER FORM.

Wrap everything in another block if you must and pass it to the partials. But only call it ONCE!

I tried to all string columns for all of the abstract columns in the import file and eventually received this error message.

Mysql2::Error: Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. You have to change some columns to TEXT or BLOBs: ALTER TABLE ‘abstracts` ADD `cy_diag_fish` varchar(255)

I have since only added those legacy fields that have data and more selectively chosen lengths and datatypes.

It would be nice if there was a way from within mysql to get the current row size so I could know how close I am.

StudySubject has many SubjectRaces StudySubject has many Races through SubjectRaces SubjectRace belongs_to :race, :primary_key => “code”, :foreign_key => “race_code”

Now race_ids returns race CODES, but assigns with race IDS?

irb(main):028:0> s.race_ids
  (0.4ms)  SELECT `races`.code FROM `races` INNER JOIN `subject_races` 
   ON `races`.`code` = `subject_races`.`race_code` 
   WHERE `subject_races`.`study_subject_id` = 14485
=> [999]
irb(main):029:0> s.race_ids = [998]
 Race Load (0.4ms)  SELECT `races`.* FROM `races` 
  WHERE `races`.`id` = 998 LIMIT 1
ActiveRecord::RecordNotFound: Couldn't find Race with id=998

I’m guessing that the key cause of this is the “through”

(For some reason the above leading “‘” translate into “%x” ???)

As of ruby19 @1.9.3-p392_1+doc, the +nosuffix option is no longer available via MacPort.

Attempting …

sudo port install ruby19 +nosuffix

will work, but won’t remove the “1.9” suffix. Now another port, “ruby_select” will be installed to do this. To make ruby 1.9 the default ruby, the following command will need to be executed…

sudo port select --set ruby ruby19

As of 20141203, we are using ruby 2.1.5

As of 20150117, we are using ruby 2.3

DO NOT USE Date.today, unless the following is what you want.

It will compute a Date and disregard the time zone.

In rails, this means that occassionally Date.today == Date.tomorrow or Date.today == Date.yesterday, depending on what time it is.

The Rails methods of Date.tomorrow and Date.yesterday reference Date.current, which uses the Time.zone, making them more accurate.

I think that DateTime.now and Time.now also ignore the timezone.

Use DateTime.current or Time.zone.now

I upgraded from Lion to Mountain Lion and things went south.

Mountain Lion no longer has the Web Sharing option in the System Preferences so starting and stopping apache by hand is required.

sudo apachectl start|stop|restart

Noticed that even though starting apache didn’t complain at the command line, the console was shouting …

Apr 23 14:17:14 fxdgroup-169-229-196-225 com.apple.launchd[1] (org.apache.httpd[1417]): Exited with code: 1
Apr 23 14:17:14 fxdgroup-169-229-196-225 com.apple.launchd[1] (org.apache.httpd): Throttling respawn: Will start in 10 seconds

Some of the conf file syntax has changed and if is invalid apache won’t say anything so check the syntax with …

sudo apachectl -t

XCode command line utilities will need reinstalled. Open XCode, THE NEW VERSION not the old yet still also installed /Developer/Applications version. The Xcode that should by in /Applications.

Preferences > Downloads > Command Line Tools

MacPorts will probably all need reinstalled.

sudo port selfupdate
sudo port upgrade outdated

Gem may also require reinstallation if they were compiled with native stuff.

I also get the following message every time that I sudo something.

dyld: DYLD_ environment variables being ignored because main executable (/usr/bin/sudo) is setuid or setgid

It goes away if I stop setting LD_LIBRARY_PATH, but I NEED THAT TO BE SET FOR OTHER THINGS. I don’t completely understand what it has to do with sudo and neither does anyone else apparently, but I WILL figure it out.

Given a file containing nothing but a list of quoted icf master ids .…

File.open('icf_master_ids_20130515.txt'){|f| while(line = f.gets); IcfMasterId.create!(:icf_master_id => line.chomp.gsub(/"/,'')); end }

I locally installed Validator S.A.C. This requires copying (and possibly modifying) a configuration file to … (perhaps another place, depending on your installation)

/opt/local/apache2/conf/

Also need to ensure that its included

I put … in my /opt/local/apache2/conf/httpd.conf

Include conf/validator-SAC.conf

Sometimes, the installation of the mysql gem doesn’t work with the defaults. It may need a little help. Sometimes it can’t find the mysql_config file. Other times it can’t find the mysql dir. And other times, it can get bad info from the mysql_config. Check out the error logs and even try running mysql_config to see what its output is.

gem install passenger (don’t put in Gemfile, for some reason)

Testing passenger locally

(first time, it takes a few minutes as is compiling.)
(creates ~/.passenger/ containing compiled code.)
(after that, C_INCLUDE_PATH doesn't need set.)
setenv C_INCLUDE_PATH /opt/local/include
passenger start

Create and edit .. nginx.conf.erb passenger-standalone.json

To run on port 80, MUST sudo

Is this “poodle vulnerable”? ssl_protocols can be controlled in nginx.conf

Seems Gary found a fix for the poodle vulnerability making the usage of Passenger unnecessary.

(poodle was that ssl vulnerability)

Add passenger to Gemfile

gem "passenger"

bundle install

bundle exec passenger start

Could it really be that simple. I had to rename nginx.conf.erb and passenger-standalone.json, but yes.

No more …

script/rails server -d -p 3000 -e production

Using rake sunspot:solr:reindex is now a redirect to sunspot:reindex and DOES NOT PASS THE ARGUMENTS on. Using rake sunspot:reindex instead.

DON'T rake sunspot:solr:reindex\[,Abstract\] as is the same as just rake sunspot:solr:reindex

DO bundle exec rake sunspot:reindex\[,Abstract\]
DO bundle exec rake sunspot:reindex\[,Sample\]
DO bundle exec rake sunspot:reindex\[,StudySubject\]

The web server no longer starts automatically on my Mac so .… (be certain to use the correct version, perhaps /opt/local/apache2/bin/apachectl )

sudo apachectl start

Apache is unnecessary unless using https.

Although, without it app can’t use the local html validation app.

And we’ve changed to use mariadb, so start that manually as well …

sudo /opt/local/share/mariadb/support-files/mysql.server start

Testing will require a live internet connection or you’ll see a lot of …

SocketError: getaddrinfo: nodename nor servname provided, or not known

If sunspot is not running, you’ll see a LOT of …

Errno::ECONNREFUSED: Connection refused

Added a “sleep” after starting sunspot as it just seems to take a moment.

bundle exec rake db:drop:all ; bundle exec rake db:create:all ; bundle exec rake db:migrate ; bundle exec rake db:test:prepare

c ; bundle exec rake sunspot:solr:stop RAILS_ENV=test ; /bin/rm -rf coverage* ; bundle exec rake sunspot:solr:start RAILS_ENV=test ; sleep 180 ; bundle exec rake ; mv coverage coverage-one-run ; open coverage-one-run/index.html ; bundle exec autotest -v --no-full-after-failed

DO NOT KEEP OLD FILES IN DB or DB/MIGRATE/. They will be included when the test db is migrated.

gem uninstall --executables --ignore-dependencies --all
gem install bundler
cd  /my/ruby/clic; bundler install
cd  /my/ruby/odms; bundler install

Unfortunately, killing the server could result in lost data that may be cached in memory. At least in theory. I don’t think that rails really caches like that.

Currently having some issues with rails 4.2.0 on production. No response in browser and requests don’t show in log. Its acting as if nothing is even there. Very confused? Going to try updating rubygems and reinstalling all gems. Dev has gem 2.2.2 (now 2.4.6) and production only has 2.0.3. Not sure if I can on production as don’t have root priveleges. Nope. Can’t update rubygems on production.

gem update --system
gem pristine --all

Maybe try brand new empty rails app if no joy? no joy for new empty rails 4.2.0 app

dump_database
bundle exec rake app:data:dump_to_csv
cd /my/ruby/odms
bundle exec rake sunspot:solr:stop
kill -9 `cat tmp/pids/server.pid`
git pull
bundle install
bundle exec rake db:migrate

bundle exec rake sunspot:solr:start
bundle exec passenger start

ssh -L 3000:localhost:3000 cube
ssh -L 3000:localhost:3000 o
http://localhost:3000

~/dump_database
cd /home/app_odms/odms/
bundle exec rake app:data:dump_to_csv
sudo /sbin/service httpd status
sudo /sbin/service sunspot-solr stop
sudo /sbin/service passenger-odms stop
git pull
bundle install
bundle exec rake db:migrate
sudo /sbin/service sunspot-solr start
sudo /sbin/service passenger-odms start

Copyright © 2011 [Jake Wendt], released under the MIT license