PurplePixel

I am testing conversion from vBulletin 3.8.11 but BB Code does not seem to convert. Am I missing any settings or this is the normal behaviour?

Thank you @CodingJungle, looks really good. I will test it. It's been frustrating not having a proper way of debugging code and if the codebase would have used some class in order to eval the code instead direct eval() that would be easy to hook up to and do this.

@Stuart Silvester It probably avoids only for the themes. But not for plugin and application hooks as they are generated on-the-fly. An it's not the only code that is generated on the fly. Search for eval( in the source code so you can see how many times you will find this.

Debugging eval() issues (hooks, etc.) are very hard since the code is loaded in the memory, so I wrote a class that can write in the /tmp/ directory the php eval'd code so one can use xdebug on the code. Moreover, it does search for the namespace and the first class name and sets the file names based on that (for the ones that do not have a class insides the file name will be sha1($code)). I have shared this patch in hope you can implement something similar for a better debugging mechanism. It can be imported with git so you can verify it.

It would be great if the core would use a class like EvalDebug::eval($code) instead of eval($code) which is impossible to modify to write to disk (and believe me I really tried to do that). Even if the EvalDebug class does not contain anything else except eval($code) inside. Like this:

class _EvalDebug
{
    public static function eval(string $code)
    {
        return eval($code)
    }
}

This way, developers can manipulate the mechanism of eval(). Let me know what you think, thanks!

eval_disk_writer.patch

15 hours ago, Paul E. said:

Eeeeek. Change that 0777 to the constant IPS_FOLDER_PERMISSION.

 

Will do. By default it's just mkdir() and that's what php.net says:

So everywhere in the code it's actually 0777 set since the $mode parameter is not specified 😄

15 hours ago, Paul E. said:

Perhaps converting the bmp images using gd/imagemagick would be a better approach during the actual vBulletin converter run?

I will try and see if it worth the effort. I think during conversion some image manipulation is happening actually.

15 hours ago, Paul E. said:

Communicate this to IPS by opening a support request and provide a link here so it isn't missed.

Thanks, will do that.

Thanks @Ryan Ashbrook and @Paul E. for advices will check what fits best for us and get back to you if I need more information. 🙂

I have found a few bugs which I have fixed through hooks. It would be great if you fix them in the forum directly. Here's a list:

#1

// File: system/Theme/Advanced/Theme.php
// From:
...
	protected static function _writeThemePathDirectory( $app, $container, $path )
	{
...
			if ( ! @mkdir( $dirToWrite ) )
...
	}
...

// To:
...
	protected static function _writeThemePathDirectory( $app, $container, $path )
	{
...
			// This fixes recursive directory creation as some applications seem to need.
			if ( ! @mkdir( $dirToWrite, 0777, true ) )
...
	}
...

#2

// File: system/Image/Image.php

// From:
...
	public static function create( $contents )
	{
...
		$signatures = array(
			'gif'	=> array(
				'47' . '49' . '46' . '38' . '37' . '61',
				'47' . '49' . '46' . '38' . '39' . '61'
			),
			'jpeg'	=> array(
				'ff' . 'd8' . 'ff'
			),
			'png'	=> array(
				'89' . '50' . '4e' . '47' . '0d' . '0a' . '1a' . '0a'
			)
		);		
...
	}
...

// To:
...
	public static function create( $contents )
	{
...
        $signatures = array(
            'gif'	=> array(
                '47' . '49' . '46' . '38' . '37' . '61',
                '47' . '49' . '46' . '38' . '39' . '61'
            ),
            'jpeg'	=> array(
                'ff' . 'd8' . 'ff'
            ),
            'png'	=> array(
                '89' . '50' . '4e' . '47' . '0d' . '0a' . '1a' . '0a'
            ),
			// Some vBulletin images have this signature.
            'bmp'   => array(
                '42' . '4d' . '36' . '4f' . '00' . '00' . '00' . '00',
            )
        );
...
	}
...

I managed successfully on setting up the conversion data and start the conversion but I cannot see any of the imported forum in the front-end. I tried to figure it out and discovered that forums do not have permissions set, so that is why.

Is there any way to automate permissions to be set to visible on all imported data? Or at least a fast way to mass-edit the forums? We have a few hundreds...

Are there tests suites available for the core Invision code and functionality?  We would like to know if our plugins and app potentially break key functionality in an automated way.

Thanks @Paul E. and @Stuart Silvester for the great feedback you both shared! I can see you both are very experienced in migration and all the advices you wrote will be added to our todo before migration list. It's like war: expect for the worst and hope for the best.

13 hours ago, PurplePixel said:

Our plan is to start with a small sample of the old solution (1500 members / posts / etc.) and iterate the conversion process until we are happy with it.

I think I wasn't very clear about this part. I was trying to say that this is my local database that I am iterating with such sample so I can see what fails and how to improve fast. When the sample works as desired, we can start testing the conversion on the whole database, then see what will fail, benchmark it, profile it and improve it.

After we are happy with this part, I think we can create on a temporary server with the real data for beta testing. Server where our team and forum members can test the solution.

13 hours ago, Paul E. said:

Don't overrely on your members though, and encourage them to report anything that seems off.

That's a great piece of advice. We have to instruct our members to report anything, not only things that are clearly off.

13 hours ago, Paul E. said:

And! Don't forget to test e-mails and make sure that test e-mails don't go to your members.

That is what I had in mind, either /dev/null, or a fake smtp catcher so we can also see if the communication is working and looking as desired.

13 hours ago, Paul E. said:

There is a constants.php setting you can set to ensure that e-mails go to /dev/null until you're ready for that.

I think you are talking about the EMAIL_DEBUG_PATH constant, which can write the outgoing e-mails to disk instead of sending them. Sounds like a good approach since we can use /dev/null while testing and write to disk on a later stage.

12 hours ago, Stuart Silvester said:

I would also recommend not making knee-jerk reactions to comments your members make immediately after the conversion. People generally do not like change and will be saying "make this work like vBulletin". Take some time and let them settle in with how the new platform works and then collate their feedback.

Yes, I am pretty sure there will be a lot of members that will be reluctant to changes and will probably complain a lot but for sure we will respect our community since it's the heart of the project.

12 hours ago, Stuart Silvester said:

A note I made when working on the performance changes says "[vBulletin 3/4] Posts step was around 900ms per 200 posts instead of 19 seconds".

That sounds like a great improvement! I suppose this tweaks exist in Invision Community 4.5.0 beta 9, right? We did not upgrade to latest yet, but will upgrade in the next few weeks.

@Stuart Silvester is there a unit testing suite for the whole application? We were thinking to create some unit testing for the plugins we are developing and I was wondering if we can integrate in something that already exists or we should create our own system.

I have to say, I am still thinking to adapt the code to a command line interface and run the forum migration from there. If feels much safer and more controllable than doing this through requests to a web service. And some existing unit tests would be really great so I can make sure the migration tool works as desired.

@Paul E. did you and your team implement an automated unit tests suite or all the tests were manually done?

Thank you all for the great feedback you provided! It really helps to see that others did conversion with millions of posts and they worked (although not perfect, but with a bit of sweat, it was done). All of this input really helps!

Thank you so much for this detailed feedback @Paul E.! I am pretty sure there are going to be some hard working days until we get the conversion right. I hope we won't miss any parts, but you never know on such complex migrations until it's too late. And then it's impossible to revert and very hard to migrate the missing data. Anyway, your feedback really helps!

As for migration taking 4-5 days, that's a lot. All this time, the forum was down or could be partially used somehow?

I also plan a profiling session to see what I can optimise in the migration scripts. Also we have to see how to plan to migrate the assets apart from database.

Our plan is to start with a small sample of the old solution (1500 members / posts / etc.) and iterate the conversion process until we are happy with it. After that, we will make a clone of the live solution which we will convert (and optimise the process even further if possible), then get 50 members from the live forum to test the data integrity apart from our team. I hope that will be enough to ensure the data integrity.

Any insights of issues with the migration process really help to not repeat them again 🙂 

37 minutes ago, Stuart Silvester said:

You would be better to use an application and store your converter files within that application rather than writing a plugin to load them from elsewhere (it's nicer to keep them in their own location than throw them in another (IPS) application). You will still need to add a hook to the software() method to define your custom converter location.

I wasn't planning to create an application, but wasn't sure what was the purpose of the two buttons in converter view. I guess I'll just ignore them 😄. So I already created a plugin that hooks to the software() method, that works well as I see and I think that is the cleanest approach. Thanks!

40 minutes ago, Stuart Silvester said:

Doing a conversion via AdminCP is perfectly fine, command line conversions are not supported.

I see. So I should trust ACP converter even for millions of posts as you say. One question though: do you have any benchmarks on how much time should I expect for conversion for 1 million posts?

Other than online doc that is available, are there any other resources for plugin and app developers?
 

Can we get official workshops/training sessions? I think we are happy to pay for a session as well.

Can we get access to Invision's repo and make PRs for fixes/tidying?

Eg: Wrong description from my point of view, because it is thrown when a dataset is empty as well.

14 hours ago, Paul E. said:

You may be overthinking this, but... don't let me dissuade you. 

I guess you are right @Paul E.. I'm a perfectionist and try to do this as clean as possible, but after all this will only happen once.

The forum has a few million posts forum and I just want to make sure I am in control when this will be done. Also I am trying to do this using command line since ACP does not feel right. Any ideas if anyone did this before? I tried to scrap the developers forum but did not find any answers.

Also @bfarber is there any way to contribute to the IPB code? I have a few ideas that may help everyone. Thanks!

Thank you for your answer @bfarber! Will the converter class be included automagically? Because for the moment I have overrider the IPS\convert\_Software::software() method through a plugin in order to make the converter show. I am trying to adapt the vBulletin 3.8.xx/4.x migration to my needs as we have custom data in our previous solution.

I am trying to create a custom conversion class and I am not sure what is the right approach. I have seen these two buttons inside the ACP conversion view when I am in developer mode:

And I have used them to create Library and Converter class files, but I don't understand how they hook with the converter or with IPS framework.

Also, is there any specific path I need to create these files into in order to be included?

I tried searching for documentation related to these buttons and the classes they create, but did not seem to find any.

Thank you!

Great idea! Is there any command line tool that can perform this at this point?

Also, is there any command line tool that can perform any tasks except the cronjob?

@bfarber getting the same issue - please see below

The following conditional fails in `system/Theme/Theme.php` - line 4882

// create the plugin template
if( \IPS\IN_DEV AND $data['app'] === 'core' AND $data['location'] === 'custom' AND $data['path'] === 'front' AND isset( $data['plugin'] ) )
{
	$plugin = \IPS\Plugin::load( $data['plugin'] );

	\file_put_contents( \IPS\ROOT_PATH . "/plugins/{$plugin->location}/dev/css/{$data['name']}", $data['content'] );
	@chmod( \IPS\ROOT_PATH . "/plugins/{$plugin->location}/dev/css/{$data['name']}", \IPS\IPS_FILE_PERMISSION );
}

Why? When you do manual upload the variables are set incorrectly.

`applications/core/modules/admin/applications/plugins.php` - line 681

$class::addCss( array(
  'app'			=> 'core',
  'location'	=> 'front',
  'path'		=> 'custom',
  'name'		=> $xml->getAttribute('filename'),
  'content'		=> base64_decode( $xml->readString() ),
  'plugin'		=> $plugin->id
), TRUE );

Please look at `location` and `path`.

@bfarber has this been resolved? - hidden but required fields are still required even though the user can't see them

I'm definitely supporting this - standardising IPB would make development and even contribution much simpler. I understand there are many challenges to make this possible but it would have compounding benefits in the future.

Hey all, 

I'd like to add an additional variable to a Form Helper - Ideally I would like achieve this using a plugin only.

Example:

Making the  `IPS\Helpers\Form\Number` to be responsive as the input currently have the `ipsField_short` class hardcoded for some reason making the handling of this field quite difficult. It's completely out of sync with the rest of the form helpers/elements.

Approach is with a New Plugin:

- Code Hook into `IPS\Helpers\Form\Number` to replacing the `html` method adding `$this->options['class']` to the very end (making it compatible with official version)

- Same Code Hook redefining `$defaultOptions` with additional class element.

- Theme Hook to `core/global/forms/number.phtml` selecting the `<input>` and completely replacing it with my own. Basically the same markup but replacing `class='ipsField_short'` with `class={$class}`. 

This makes it completely compatible with the original Number field but I can actually change the class of the number input field. My question is: how can add the additional class variable/parameter to `<ips:template parameters..` with a plugin? 

Thanks in advance

2 hours ago, Morrigan said:

Well the text is likely translatable in the Language.

As far as the look of the labels:

[data-pagecontroller="register"] .ipsFieldRow_label {
	Change the look here.
}

 

Thank you Morrigan. This is what I ended up doing. I was just hoping there was a way to actually remove things instead of using css to hide it from the form. But I think in this case it will be ok. 

Hi all, I'm trying to find where or how I can update the registration page? I've found some bits and pieces but specifically I want to add classes to change how the labels of the form look, like where it says "Display Name" etc https://gyazo.com/ed1f84250eb122cafe2185ef6adb0c78.png

Also I'd like to remove the "REQUIRED" span.

Can anyone point me to the template? I've tried the popupRegisterTemplate.phtml and register.phtml as well as a few others but can't find where these come from. Thanks!