Make IPB more developer-, version control- and testenvironment -friendly

[TSP]

Hi, we have a couple of suggestions and would like to discuss how IPB could be made more developer-friendly. We are currently responsible for three different large IPB-forums that all share the same type of setup.



We have a setup where we:

• Have 6 app servers the code is distributed on (prod)
• 2 frontend servers/varnish (prod)
• Test and stage environment, two different servers
• Deploy-scripts for deploying git commits/branches/tags to prod, stage or test.
• Use a shared file volume for the folders public, cache and uploads.
• Modified conf_global.php file that sets the board url, and database information to be different based on the server you access it from. Thus we're able to have test-areas like stage.example.com and test.example.com .
  

We use the test area with it's own test database (from some recent backup) and it uses it's own versions of the folders cache, public and hooks, to do test upgrades on etc. The stage area is on a shared database with prod and is mostly used to test custom code changes or bug fixes before we deploy them to prod.

File paths to the upload, hooks and cache-folders to the shared file volume (for stage and prod) are set up by using symlinks from the app-servers for stage and prod. On the test-server it's just using the normal folders. (So they are not shared with prod and stage)



Quick overview over suggested changes (In prioritized order)

• Make path to the cache-folder configurable in conf_global.php
• Move 4 "static" php-files and 1 html-file from the cache/skin_cache-folder (only autogen-files should be in cache-folder)
• Use a script or tool to ensure that all files in releases have unix line endings
• Move or make paths to folders with autogenerated files in folder public configurable from conf_global (style_css etc.)
• Make path to upload-folder configurable in conf_global.php
  

There are a couple of problems with this setup and I believe IPS could do some improvements to IPB that would make it more friendly to server environments like this one.

The cache-folder contains a couple of files that are not generated:
cache/skin_cache/emailWrapper.php
cache/skin_cache/ipsDriverError.php
cache/skin_cache/ipsServerError.php
cache/skin_cache/memberMap.php
cache/skin_cache/settingsEmpty.html

We believe they should be moved to another folder than cache, since they are "static" php-files that need to be uploaded again between every version to ensure they are the newest one. With our current setup we have to ensure these files are copied into the folder manually at every upgrade. If this could be changed we wouldn't have to worry about it and could "forget" the folder more easily since the whole cache-folder would only contain auto-/ACP-generated files. (and index.html's)

The public folder. Not really sure which folders files are generated to, but I believe they are style_images, style_css and style_emoticons. Could you possibly move these folders to another "core"-folder or could you let us define the file paths to all the folders that contain auto-generated files in the conf_global.php? Could you also let us define the file path for the public folder as a whole/for content other than autogenerated?

We would like to be able to have our public-folder, the javascript etc. as part of our git repo, so we don't have to copy all of the files manually to our shared file volume at every upgrade either and easier to edit the code. That would also make it easier to deploy different versions of IPB to our stage and test setups.

Upload-folder. Could you let us define the file path to this in conf_global.php, should override ACP path if set.

Could you also make sure line endings are in the same unix format consistently in all the files you distribute for release? Currently it seems that your code files are randomly either dos or unix, but most of them are in unix line endings. This causes problems when doing git diff on new IPB code in some cases where the type of line endings have changed. Since Git will report all the content/lines in the file as fully removed and then fully added, even though just some lines changed.


Just some changes I feel would make these kind of setups more supported, especially the cache-folder file path change.

[eGullet]

Those are all excellent suggestions: the line ending one in particular seems like a no-brainer, I'm surprised Invision's own internal source control software doesn't give them fits over this. I was under the impression, however, that you could already specify the upload folder in conf_global.

[Marcher Technologies]

PM'ed you with a tool I made and use for the line-break issue TSP.... made for a completely different reason(win->linux OS swap fix) but you should find it useful.

[TSP]

I almost forgot one suggestion here.

Cookie domain should be configurable from the conf_global.php and should override the one in ACP if set.

[Marcher Technologies]

I almost forgot one suggestion here.

Cookie domain should be configurable from the conf_global.php and should override the one in ACP if set.

unless i am miles off here, this could already be done.
anything defined in $INFO is loaded into ipsRegistry::$settings, so use it and empty the acp val.

[Marcher Technologies]

unless i am miles off here, this could already be done.

anything defined in $INFO is loaded into ipsRegistry::$settings, so use it and empty the acp val.

Actually, scratch that, you shouldn’t even need to empty the acp val, $INFO is honoured over all AFAIK, otherwise setting power settings here would not work proper.

[TSP]

Actually, scratch that, you shouldn’t even need to empty the acp val, $INFO is honoured over all AFAIK, otherwise setting power settings here would not work proper.

Should that work for all settings? I discovered that our stage area is not having trouble because of the cookie domain setting, but instead having trouble due to not being on a domain where we have https. We have https enabled from the ACP, thus not possible to log in on our stage area. Since it shares the database with prod, we can't change the setting in the ACP. I tried to add $INFO['logins_over_https'] = '0' to the stage-part of our config, as that's the setting key from www/admin/applications/core/xml/core_settings.xml, but that didn't do anything appearantly.

[TSP]

Quick overview over suggested changes (In prioritized order)

• Move 4 "static" php-files and 1 html-file from the cache/skin_cache-folder (only autogen-files should be in cache-folder)
• Use a script or tool to ensure that all files in releases have unix line endings
• Move or make paths to folders with autogenerated files in folder public configurable from conf_global (style_css etc.)
• Easily let us override settings defined in the ACP from a php file.
  

Just some changes I feel would make these kind of setups more supported, especially the cache-folder file path change.

Just updating this slightly.

For example, we could have a php-file called settings_override.php (similar to constants.php) where you could go:
$settings['logins_over_https'] = 1;
$settings['ipb_cache_path'] = '/path/';

And what was set there would be used instead of what the value was set to be in the database/ACP.

In order to keep people from forgetting about the file, you could simply have a message on top of the system settings area like this: We've detected that the settings_override.php file have been created by you in your forum root folder. Please check this file if you're experiencing problems when updating any of your settings from the Admin CP.

This would be really really helpful! :smile:

[Wolfie]

We have a setup where we:

• Use a shared file volume for the folders public, cache and uploads.
• Modified conf_global.php file that sets the board url, and database information to be different based on the server you access it from. Thus we're able to have test-areas like stage.example.com and test.example.com .
  

We use the test area with it's own test database (from some recent backup) and it uses it's own versions of the folders cache, public and hooks, to do test upgrades on etc. The stage area is on a [u]shared database[/u] with prod and is mostly used to test custom code changes or bug fixes before we deploy them to prod.

Should that work for all settings? I discovered that our stage area is not having trouble because of the cookie domain setting, but instead having trouble due to not being on a domain where we have https. We have https enabled from the ACP, thus not possible to log in on our stage area. Since it [u]shares the database with prod[/u], we can't change the setting in the ACP.

You should NEVER have two locations sharing the database or files. That can and eventually will lead to problems. Trust me, keep it all separate.

As for a modified conf_global, you can do that yourself. Create a folder named "conf_globals" and inside that create files that are customized for each possible URL you would be accessing and then in the primary file, have it check the URL being accessed and then require() the appropriate file to import the desired settings. conf_global.php is a PHP script and you can do things like that to it. However, going back to the point of not sharing stuff... Bad idea.

Only possible exception I could see would be using the same set of files for multiple installs (provided you have a valid license for each that is). Then based on the URL, have a unique 'conf_global.php' and 'constants.php' file for each so that content/files unique to each URL wouldn't interfere with the other URL's. But that would only be after some thorough testing to make sure it doesn't break or cause problems.

Quick overview over suggested changes (In prioritized order)

• Make path to the cache-folder configurable in conf_global.php
• Move 4 "static" php-files and 1 html-file from the cache/skin_cache-folder (only autogen-files should be in cache-folder)
• Move or make paths to folders with autogenerated files in folder public configurable from conf_global (style_css etc.)
• Make path to upload-folder configurable in conf_global.php

Any path configurations (or recommended path configurations) should be done via the 'constants.php' file. Open up 'initdata.php' and you can see what you can set from within 'constants.php' by seeing what values are checked before being set. (You can use 'constants.php' to set the ACP and hooks folders, for example.) Being able to customize the location of the 'cache' folder would be nice though.

I'd like to toss in the ability to specify the path and name of the 'conf_global.php' file from within the 'constants.php' file. Not only increase security a little, but would make multiple installs more possible & reliable.

[TSP]

You should NEVER have two locations sharing the database or files. That can and eventually will lead to problems. Trust me, keep it all separate.

The idea for a stage area is to test code changes (things that are not done with hooks) on the most recent database and with the most recent files to ensure things are not breaking apart before deploying the changes to production. Can also be used to test that everything looks ok after upgrades, before you take your downpage away.

As for a modified conf_global, you can do that yourself. Create a folder named "conf_globals" and inside that create files that are customized for each possible URL you would be accessing and then in the primary file, have it check the URL being accessed and then require() the appropriate file to import the desired settings. conf_global.php is a PHP script and you can do things like that to it. However, going back to the point of not sharing stuff... Bad idea.

Not sure what you're referring to here, we already have the conf_global files set up like we want them to be.

Example:

error_reporting(E_ALL & ~E_DEPRECATED & ~E_NOTICE);

if (file_exists("/etc/area") && strpos(file_get_contents("/etc/area"), "test") !== false) {

    $INFO['board_url']                  =       'http://test.example.com';

    $INFO['sql_host']                   =       'mysql-test.example.com';

        $INFO['sql_pass']                       =       'password-test';

        $INFO['sql_user']                       =       'user-test';


}

else if (file_exists("/etc/area") && strpos(file_get_contents("/etc/area"), "stage") !== false) {

    $INFO['board_url']                  =       'http://stage.example.com';

    $INFO['sql_host']                   =       'mysql-prod.example.com';

    $INFO['sql_user']                   =       'user-prod';

    $INFO['sql_pass']                   =       'password-prod';

}

else {

    $INFO['board_url']                  =       'http://www.example.com';

    $INFO['sql_host']                   =       'mysql-prod.example.com';

    $INFO['sql_user']                   =       'user-prod';

    $INFO['sql_pass']                   =       'password-prod';

}

** rest of the conf_global settings **

Only possible exception I could see would be using the same set of files for multiple installs (provided you have a valid license for each that is). Then based on the URL, have a unique 'conf_global.php' and 'constants.php' file for each so that content/files unique to each URL wouldn't interfere with the other URL's. But that would only be after some thorough testing to make sure it doesn't break or cause problems.

Not something we plan on doing. This is for stage, test and prod area of the same board, used for testing. It's not used as different boards, it's the same board, just different versions of it, you could say, only used for testing and developer purposes for that specific board.

Any path configurations (or recommended path configurations) should be done via the 'constants.php' file. Open up 'initdata.php' and you can see what you can set from within 'constants.php' by seeing what values are checked before being set. (You can use 'constants.php' to set the ACP and hooks folders, for example.) Being able to customize the location of the 'cache' folder would be nice though.

We do that already. The path suggestions I mention here are configurable from the ACP already, but can not be overwritten from initdata.php/constants.php or conf_global. We would like it to be possible to override the paths defined in the ACP from a configuration file. That way we can handle the logic so we can point the uploads-folder (for example) for the test version of the board to a different location. Then we wount have to use symlinks or having to remember to change such things from the ACP on the test version every time we send a new database dump to it.

[Wolfie]

The idea for a stage area is to test code changes (things that are not done with hooks) on the most recent database and with the most recent files to ensure things are not breaking apart before deploying the changes to production. Can also be used to test that everything looks ok after upgrades, before you take your downpage away.

I just fail to see the logic in doing it that way as opposed to running a development site that has the same URL structure (just on dev.somesite.com instead of www.somesite.com) using an exact copy of the database and all the files (excluding conf_global.php and of course with any mods done to the files on the dev site). Because once you have thoroughly tested everything, it's just a matter of copying all the files over (except conf_global.php of course) and poof, live site is running with the modifications.

If the way you're doing it works for you, that's great. I just strongly recommend against sharing databases and/or files because eventually something bad will happen.

Not sure what you're referring to here, we already have the conf_global files set up like we want them to be.

Example:

error_reporting(E_ALL & ~E_DEPRECATED & ~E_NOTICE);

if (file_exists("/etc/area") && strpos(file_get_contents("/etc/area"), "test") !== false) {

    $INFO['board_url']                  =       'http://test.example.com';

    $INFO['sql_host']                   =       'mysql-test.example.com';

        $INFO['sql_pass']                       =       'password-test';

        $INFO['sql_user']                       =       'user-test';


}

else if (file_exists("/etc/area") && strpos(file_get_contents("/etc/area"), "stage") !== false) {

    $INFO['board_url']                  =       'http://stage.example.com';

    $INFO['sql_host']                   =       'mysql-prod.example.com';

    $INFO['sql_user']                   =       'user-prod';

    $INFO['sql_pass']                   =       'password-prod';

}

else {

    $INFO['board_url']                  =       'http://www.example.com';

    $INFO['sql_host']                   =       'mysql-prod.example.com';

    $INFO['sql_user']                   =       'user-prod';

    $INFO['sql_pass']                   =       'password-prod';

}

** rest of the conf_global settings **

I was thinking something along the lines of this (tweaking your example some to make it relevant to you):

error_reporting(E_ALL & ~E_DEPRECATED & ~E_NOTICE);

*** All settings here ***

if (file_exists("/etc/area")

{

  if ( preg_match( '#(test|stage)#', file_get_contents("/etc/area"), $site ) )

  {

    die( "Fatal error." );

  }

  $site = $site[0];

} else {

  $site = 'public';

}


@require( "conf_globals/{$site}.php" );

In a folder name "conf_globals/" you could then make test.php, stage.php and public.php, with all relevant configuration details inside of it. You could also add in changes for other settings (like if for one install you want to disable friendly URL's, you could have it defaulted on in the settings first, then in the custom script file, have it flagged to off).

To add in another test/development site that has a '/etc/area' available, just make a custom file for it and add the unique name (found in /etc/area) and it's added.

What I was referring to though was looking at the domain name/URL being used to access it and have it determine it that way. This goes along with what I was saying about multiple individual sites using the same files. (See below quote/reply.)

Heck, with the example I provided, you could have a full conf_global file (including details for the public side database) and have it switch to different details after. Load default values, then call in an optional script that overwrites the default values with custom values. ;)

Not something we plan on doing. This is for stage, test and prod area of the same board, used for testing. It's not used as different boards, it's the same board, just different versions of it, you could say, only used for testing and developer purposes for that specific board.

I was pointing it out as something that could be tried in general, not really aimed at what you're doing. More of less a thought related to the topic but nothing more. There are potential problems with doing it though, at least as it is now. When you run the upgrade script, I think it writes out to the conf_global file and that could cause it to become corrupt. That's a big reason why I mentioned about defining the location and name of the conf_global script, because it would bring it down to one file to truly configure with it defining what other files to look for and such.

We do that already. The path suggestions I mention here are configurable from the ACP already, but can not be overwritten from initdata.php/constants.php or conf_global. We would like it to be possible to override the paths defined in the ACP from a configuration file. That way we can handle the logic so we can point the uploads-folder (for example) for the test version of the board to a different location. Then we wount have to use symlinks or having to remember to change such things from the ACP on the test version every time we send a new database dump to it.

I see 'conf_global' as more or less being a config file that contains the database details as well as a couple of details that, if lost, can be easily rebuilt. If the file gets deleted, just a matter of making a copy and correcting the database details. Sure, you lose information like when the board was first installed, but you can make a good guess based on when the oldest account was made. So if the upgrade script does write out to it (it used to, not sure if it does anymore), then there's a risk of custom code getting lost.

I see 'constants.php' as being a pre-emptive folder/file structure folder that, if it's not there, will rely on the defaults, and so shouldn't be modified much (if at all). So I believe this should be a file that never (or rarely) gets altered, with conf_global being changed more often (such as changing the db details). That's why I mentioned about it being the file for requests for configurable details like the cache folder.

[Dmacleo]

didn't know you could share DB, thought I had read somewheres would get cache slams and table corruptions.

[TSP]

I just fail to see the logic in doing it that way as opposed to running a development site that has the same URL structure (just on dev.somesite.com instead of www.somesite.com) using an exact copy of the database and all the files (excluding conf_global.php and of course with any mods done to the files on the dev site). Because once you have thoroughly tested everything, it's just a matter of copying all the files over (except conf_global.php of course) and poof, live site is running with the modifications.

That's what the test area is used for. You don't always do large changes that completely change everything around, and sometimes you want to use the live database so it's quicker to test things.

Nowadays all our test areas run on the 3.3 branch anyway, so if we need to test small changes done on the 3.2 branch (master) then we can't do it on the test area without:
1. Dump the DB from our bacup-slave to test-mysql again, takes 2-3 hours to dump + import.
2. Switch to the master branch (in git) on the test area (Currently on upgrade33-branch)
3. Copy files from /nfs/path/public, -hooks and -cache to the /local/path/... logged in from the test server.

We could of course run stage on a second copy of the database, so they were different. But it's limited how many copies of 20GB-databases we want to have.

If the way you're doing it works for you, that's great. I just strongly recommend against sharing databases and/or files because eventually something bad will happen.

I'm aware of the problems that could arise, but we don't test any code or config changes that could cause such serious problems to occur. And no errors have occured because of this yet.

In a folder name "conf_globals/" you could then make test.php, stage.php and public.php, with all relevant configuration details inside of it. You could also add in changes for other settings (like if for one install you want to disable friendly URL's, you could have it defaulted on in the settings first, then in the custom script file, have it flagged to off).

Heck, with the example I provided, you could have a full conf_global file (including details for the public side database) and have it switch to different details after. Load default values, then call in an optional script that overwrites the default values with custom values. ;)

I was pointing it out as something that could be tried in general, not really aimed at what you're doing. More of less a thought related to the topic but nothing more. There are potential problems with doing it though, at least as it is now. When you run the upgrade script, I think it writes out to the conf_global file and that could cause it to become corrupt. That's a big reason why I mentioned about defining the location and name of the conf_global script, because it would bring it down to one file to truly configure with it defining what other files to look for and such.

Ok, I was just getting confused since I thought your replies were "direct replies" to my questions. ;)

Good idea, but you would need to be able to do a settings override on some settings defined in the ACP to get it working, and that's not currently supported it seems. (Or strictly.. you wouldn't, but it would be a lot more manageable than to have to visit all the different ACPs to change the needed settings if you had several boards running on the same code.)

I see 'conf_global' as more or less being a config file that contains the database details as well as a couple of details that, if lost, can be easily rebuilt. If the file gets deleted, just a matter of making a copy and correcting the database details. Sure, you lose information like when the board was first installed, but you can make a good guess based on when the oldest account was made. So if the upgrade script does write out to it (it used to, not sure if it does anymore), then there's a risk of custom code getting lost.

I'm quite sure it doesn't. We haven't experienced any problems when upgrading so far (and it would've only updated it on one of the six app-servers anyway). The conf_global.php file is part of our git repo so when the repo would be re-deployed after upgrades it would just send the git repo version of the conf_global-file to all the app-servers anyway.


But we need to be able to define a couple of more paths to make it properly git-friendly without using symlinks or ignoring the public-folder completely. The public-folder contains js-files that would be particularly useful to have part of our git repo. But currently it's too much work and/or symlinks needed to make it happen.

I'm refering to the following suggestions:

• Move 4 "static" php-files and 1 html-file from the cache/skin_cache-folder (only autogen-files should be in cache-folder)
• Move to cache-folder or let us configure the paths to folders with autogenerated files currently in the public-folder (style_css etc.)
  

[Wolfie]

I'm aware of the problems that could arise, but we don't test any code or config changes that could cause such serious problems to occur. And no errors have occured because of this yet.

Like I said, if it's working for you... Just I recommend against it because one little mistake can cause major problems. (Not implying/inferring/saying that you or anyone else doing it is incompetent, just mistakes can happen, so I discourage taking the risk.)

Good idea, but you would need to be able to do a settings override on some settings defined in the ACP to get it working, and that's not currently supported it seems. (Or strictly.. you wouldn't, but it would be a lot more manageable than to have to visit all the different ACPs to change the needed settings if you had several boards running on the same code.)

I was meaning that each site would still be different, just using the same core files. For example, when the config file detects you're using "somesite.com" as the URL, it would load a config for that (including database details). But if you go to "somedomain.com" then while it would be using the same core .php files, it would be using a different database. Again, requires the appropriate number of owned licenses to do that, just saying it's one way it could be done. (Reference to this is in below reply.)

But we need to be able to define a couple of more paths to make it properly git-friendly without using symlinks or ignoring the public-folder completely. The public-folder contains js-files that would be particularly useful to have part of our git repo. But currently it's too much work and/or symlinks needed to make it happen.

I'm refering to the following suggestions:

• Move 4 "static" php-files and 1 html-file from the cache/skin_cache-folder (only autogen-files should be in cache-folder)
• Move to cache-folder or let us configure the paths to folders with autogenerated files currently in the public-folder (style_css etc.)

I think it would be beneficial if all of the locations could be dynamically defined. initdata.php containing the default values, but if someone wants to go into extra steps, they could have a different structure defined, including the 'conf_global' file. In all honesty, would be nice if the details from conf_global could be added into constants.php and if no 'conf_global' file is defined, then it wouldn't attempt to load one. Still have the file there, for any code that depends on it being there, but not require anything to be inside of it.

If that were done, then running multiple sites from one set of files would be way easy. (Referencing above reply.) From within the ACP, determine the resulting URL's and with the use of .htaccess, could have both sites appearing to use the same structure (/public folder, for example), but on the server, it might actually be 'somesite/public' or 'somedomain/public', etc.

Mind you, that would be more for the fun of trying it rather than actually doing it. But it's an example of something that would be much easier if more internal (file system) locations could be defined in constants.php, such as you are requesting.