Double empty lines after upgrade. Plan to fix this?

[estan]

I did a test upgrade from 3.4.7 to 4.0.1. There were few minor issues here and there but overall all looks good.

The most major concern so far I have after the upgrade is the double lining/spacing on articles/posts etc. Many posts and articles, where an extra empty line or 2 was put intentionally to better separate paragraphs in 3.4.7, look with way too much extra space after the upgrade to 4.0.1. As I understand it this is because of the difference of the editors and how they handle paragraphs, new lines etc etc.

This is a big concern for me. Obviously, I can not go and check one by one many thousands of posts, articles etc. and fix them manually. And they definitely do not look good now. This is maybe the only thing that prevents me to upgrade now. Anyone any ideas for me? 

@Charles or @bfarber or anyone else from IPS, any thoughts or suggestions here? Any way to deal with this in a smart and automatic way?
What about if during upgrade there is an optional checkbox for automatic removal of extra empty lines? Or something like that? Or any way to deal with this after the upgrade?

 

[opentype]

 Anyone any ideas for me?

​It’s just a default spacing setting (margin/padding) for the paragraph tag (<p>). You can adjust that in your custom.css however you like. 

[estan]

Thanks for helping out Ralf!

Well, I know that I can edit the CSS. I would love to hear if there are other options on the table though.

I like the default CSS in 4.0, just do not enjoy how the upgrade is handled for empty lines and wanted to hear if there are any solutions/suggestions. Or an upgrade procedure change coming soon?

[craigf136]

I think his point was that he shouldn't need too and that it should be accounted for with the differences in editors. I've upgraded and a lot of the old posts look awful as a result, for those who are upgrading and just want to use themes/plugins/application from the marketplace and don't want to do edits etc, it's not very good.

It bascially boils down to double tap of enter/return on the old cke editor compared to the new one, it isn't handled very well in the upgrade procedure.

[opentype]

 

I like the default CSS in 4.0, just do not enjoy how the upgrade is handled for empty lines and wanted to hear if there are any solutions/suggestions. Or an upgrade procedure change coming soon?

​This has nothing to do with the upgrade routine. It’s a stylistic decision set in the default theme. Just as default colours. 

[estan]

I would love to hear if there are other guys who have the same concerns as me. 

Also, @Charles have you you guys discuss this internally and do you have a decision about this? I want to know, so I can decide when to do the upgrade of the live site. 

[opentype]

I want to know, so I can decide when to do the upgrade of the live site. 

Do you understand what I have been explaining? It isn’t like there are additional lines added during upgrade. It is a style setting in the theme — therefore it doesn’t matter when you upgrade. You can change the spacing globally and non-destructively at any time. That’s a matter of fact. 

[AutoItScript]

It's a style setting in the sense that you can add this to your custom.css in IPB4.

/* paragraph spacing same as 3.4.7 */
p {
-webkit-margin-before: 0em;
-webkit-margin-after: 0em;
} 

And yes everything will look the same as 3.4 (almost, the spacing in the ckeditor is still big, but I digress).

The issue is that IPB4 has the correct css paragraph styling and IPB3 had it wrong. In IPB4 if you want a new paragraph (i.e. the paragraph is separated by some whitespace) then you just press enter once and you get this:

<p>First para</p>
<p>Second para</p>

Perfect html.  If you want lines bunched together you can use Shift+Enter to get <br>.

On IPB3 if you wanted a new paragraph you had to press enter twice to get the correct spacing so you would end up with:

<p>First para</p>
<p>&nbsp;</p>
<p>Second para</p>

This looks "double spaced" on IPB4 so you have two choices:

Option 1: Add the CSS above and work in the same "broken" way as IPB3

Option 2: Leave the new more-correct spacing style but accept that all your old posts will look like crap. If your users are happy that old posts look weird then it's the best way to go - but it's not pretty.

What is needed is option 3:

Option 3: Leave the new spacing style but fix up old posts by removing erroneous <p>&nbsp;</p>

It's not that easy to achieve though. Once you start having posts with multiple blank lines together it's tricky to process. If IPS didn't already consider this for the upgrade routine then I doubt they will now. I'm not sure if I can be bothered to code it as a custom mod for my upgrade either

 

Edit: Something odd though, the very old posts on this site don't appear have <p>&nbsp;</p> in them. Which means that

1. For some reason, users here never did double-enter to get paragraphs (unlikely, unless IPS weren't using their own default 3.4 css)
2. Someone has run a script to remove the extra paragraphs.  And if so, can we have it please

[Tanja]

We have done a test upgrade a week ago and as far as I can see, our post have been rebuilt correctly, meaning the empty lines have been removed.

@estan Have your background rebuilding processes all finished and you still have those double line spaces?

[ram108]

​It’s just a default spacing setting (margin/padding) for the paragraph tag (<p>). You can adjust that in your custom.css however you like. 

​I had the same issue after upgrade and it was not CSS settings problem:

<p>&nbsp</p>
<p>&nbsp</p>

Used mysql commands to remove it from database.

[opentype]

​I had the same issue after upgrade and it is not CSS settings problem:

It’s the space that is added to the paragraphs, that is bothering you guys. And you can change that and remove it completely through CSS.

 

(What’s in the code, i.e. a line-break or a paragraph end, is a different discussion. And it has been discussed extensively already.)

[ram108]

you can change that and remove it completely through CSS

​Some of posts have multiple lines and others not. The original posts in old forum looks the same - without any exess lines. It is not possible (and should not) to fix it with CSS.

Better to fix converson script to have posts looks the same before and after upgrade.

[opentype]

Better to fix converson script to have posts looks the same before and after upgrade.

Seriously? Again? 

The posts are just DATA (text, html codes, bb-codes, whatever). When the page is loaded, the CSS STYLING is applied to that data. It’s at THIS POINT that paragraphs get that extra space. And that space was a stylistic decision of the theme designer. It is not coming from the “conversion script”. It’s a theme styling setting as I have said multiple times now. Just as the avatar images are rounded now as a stylistic setting. But that doesn’t mean that the conversion script has cut the images into circles and the conversion script needs “fixing” if someone wants to stick with square images. It’s just theme styling. 

 

[ram108]

Ralf, I believe it is misunderstanding. I am telling about incorrect html conversion of posts that give multiple empty paragraph tags (that have to be removed), you are telling how to hide it with CSS.

But the problem is still exist with some sites. 

[WhyCry]

​I had the same issue after upgrade and it was not CSS settings problem:

<p>&nbsp</p>
<p>&nbsp</p>

Used mysql commands to remove it from database.

​Yes, I had the same issue. Only solution is a manual mysql search&replace fix. It has nothing to do with the CSS styling as some members here claim.

[opentype]

Ralf, I believe it is misunderstanding. I am telling about incorrect html conversion of posts that give multiple empty paragraph tags (that have to be removed) …

When there are two <p> tags in 4.0, what was there in the respective 3.4 post? One line-break? Two line-breaks?

[ram108]

Sure, I don't know now. I had chaos in posts after conversions. 

 

 

Some of posts were the same like in old forum, but most of posts had double, triple and more lines between text paragraphs. It was unreal to read that messages. 

 

I rise a ticket wait for a week and made everything myself. It was not easy and I spend a day on it. Had to save a dump and analyze it manually. Because not all posts were converted with simple mysql query. Used preg replace on mysql dump to make all posts perfect.

 

It was two times I had the problem after upgrade. All forums were in Russian.

 

(empty lines just to understand the issue)

 

 

 

[Colonel_mortis]

The problem is that in IPB3.4, when you press enter, it looks
like this, with no gap between the lines. This meant that some members

added two new lines to make paragraph separations clear. Unfortunately, in IPB4, a single enter

looks like this, and having two new lines

 

looks like this. From a code perspective, the difference is just that <p> tags now have a margin where they didn't before. Having a margin on the <p> tags is a positive improvement - you have already shown that some members were implementing it themselves before this change was made. Obviously, you don't want to undo the change (though if you did, you would just need to add

p {
    margin: 0;
}

to a custom CSS file).

If IPS were to make a fix, that fix would have to take account of the fact that some people may have added more lines (perhaps they wanted a really large gap, so they added 3 new lines in the middle), so to emulate that, the parser needs to remove every other empty new line. That makes the process fairly complicated to implement, and would probably end up messing up somebody's post. It might be a better solution than the current system though, if IPS are prepared to make a change like that.

Alternatively, they could make it so it only emulates the old behaviour on old posts (by adding a .UpgradedPost class or something), but that then becomes more difficult to maintain, eg when IPB5.0 comes out and all the posts have to be upgraded again, and people will complain that their ancient posts have messed up formatting now because the old code holding it together has been removed. It would also lead to some issues when the member wants to edit their post, and suddenly by editing to fix a typo, it they have now messed up all the formatting.

You can't win, no matter what you do. The current system is non-optimal, but without putting in a fair bit of work (which could be quite expensive) to make it not mess up someone's post, they don't have a huge number of options. It would be nice if they did implement a fix, but lets be honest, all it's doing is removing a few extra blank lines. Your members can fix their posts if they are completely broken, and most of the threads that have been converted will end up being superseded by new threads made in the new editor anyway after not too long, so it's not the end of the world if a few posts have too much white space. I understand your frustration though, don't get me wrong, I just think that there are enough workarounds to make this issue less significant than a lot of the others that people have raised with the new software.

[ram108]

@stormdrive thank you for describing the issue. My English is not enought for it. 

some people may have added more lines (perhaps they wanted a really large gap, so they added 3 new lines in the middle), so to emulate that, the parser needs to remove every other empty new line. That makes the process fairly complicated to implement

It is not easy to implement but we need an option to do such conversion. Imagine more than 50'000 posts (in my case) and 60% of them are unreadable with multiple spaces that make 4 paragraph post with double screen height size. And if you have 10-50 paragraphs post ((( it will be crazy to read it... And double crazy to fix it manually.

[AutoItScript]

I going to run this against my posts in this 4.0.2 test upgrade to see if it looks ok. I'm just duplicating bits of RebuildPosts.php as a new background task that I can queue up.

// First change double lines in single lines and protect them
$item->$contentColumn = preg_replace('#<p>&nbsp;</p>\s*<p>&nbsp;</p>#', '<p class="doublespace-processed">&nbsp</p>', $item->$contentColumn);

// Now remove remaining single lines
$item->$contentColumn = preg_replace('#<p>&nbsp;</p>#', '', $item->$contentColumn);

// Now restore the double lines as single lines
$item->$contentColumn = str_replace('<p class="doublespace-processed">&nbsp</p>', '<p>&nbsp;</p>', $item->$contentColumn);

It could be neater code (no doubt there's an all-in-one regex  ), but basically it will change:

a single <p>nbsp;</p> into nothing
a double <p>nbsp;</p> into a single <p>nbsp;</p>
3 x <p>nbsp;</p> into 1 x <p>nbsp;</p>

And so on.

 

[opentype]

And if you have 10-50 paragraphs post ((( it will be crazy to read it... And double crazy to fix it manually.

​What’s so crazy about  just setting p { margin: 0; } ? It faster done than writing a post in this topic. 

[estan]

Thanks for everyone contributing different thoughts and solutions. This really helps me.

@Tanja, to answer your question, the rebuilding process for my test upgrade is only about 30% done. However, the topics/posts I checked that now have the correct formatting, still have the empty lines. So, unless this empty spacing thing is getting fixed at the end of the upgrade routine, I would say, it is not ok for me.

@AutoItScript, the 3 options your mentioned, along with the detailed info, is exactly what I wanted to know. Please tell me a bit more how do you do this:

I going to run this against my posts in this 4.0.2 test upgrade to see if it looks ok. I'm just duplicating bits of RebuildPosts.php as a new background task that I can queue up.

// First change double lines in single lines and protect them
$item->$contentColumn = preg_replace('#<p>&nbsp;</p>\s*<p>&nbsp;</p>#', '<p class="doublespace-processed">&nbsp</p>', $item->$contentColumn);

// Now remove remaining single lines
$item->$contentColumn = preg_replace('#<p>&nbsp;</p>#', '', $item->$contentColumn);

// Now restore the double lines as single lines
$item->$contentColumn = str_replace('<p class="doublespace-processed">&nbsp</p>', '<p>&nbsp;</p>', $item->$contentColumn);

It could be neater code (no doubt there's an all-in-one regex  ), but basically it will change:

a single <p>nbsp;</p> into nothing
a double <p>nbsp;</p> into a single <p>nbsp;</p>
3 x <p>nbsp;</p> into 1 x <p>nbsp;</p>

And so on.

 

​Can you provide me instructions how to test this by myself? Not sure how to create a new background task and run it.

Also, can I first upgrade to 4.0.2 fully and then if I decide later ... to run this suggested background task that you described above? Or should this be most necessarily done during upgrade itself?

 

[opentype]

However, the topics/posts I checked that now have the correct formatting, still have the empty lines. So, unless this empty spacing thing is getting fixed at the end of the upgrade routine, I would say, it is not ok for me.

If a person has created an empty line in a post, than of course that empty line will be preserved and the spacing will appear however it is set in the theme. That’s an expected behaviour. The upgrade routine shouldn’t apply destructive changes to the posts, just to target certain settings of the default theme. 

Also, can I first upgrade to 4.0.2 fully and then if I decide later ... ​

After the upgrade and after the background tasks have run. Note, that these destructive procedures. You cannot revert them. 

[Colonel_mortis]

​What’s so crazy about  just setting p { margin: 0; } ? It faster done than writing a post in this topic. 

​The problem with doing that, at least in my opinion, is that it's undoing a change which, for the most part, is for the better - it may not be ideal for posts that have been upgraded from IPB 3.4, but for all new posts, the distinction between paragraphs and just basic line breaks is a positive change - it means that people have to rely on using double line breaks to achieve the formatting that they want far less often, making it better both in terms of the user experience and more semantically correct (not that semantics are hugely important, but still).

The worst that the destructive conversion method could do, as long as it performs it's fit for purpose (which I see no reason for it not being), is that too much white space is removed, so posts have slightly less white space than they should. If the posts are unreadable now because the members on that forum made excessive use of white space, it's probably worth it to reduce the amount of blank lines for a very small risk that some posts may have too many lines removed (but, notably, will still be guaranteed to have some space between the paragraphs because all paragraphs have a built in margin now.

[opentype]

… the distinction between paragraphs and just basic line breaks is a positive change … 

​I personally agree and I use it this way on my new, not upgraded 4.0 site, but the people here complaining about this (and their community users) probably just want everything to work like it has always worked. And setting the margins to zero does exactly that. It “fixes” the old posts and lets all users write their posts like they used to. Better code, same look and handling. 

Going another route, e.g. to change the old posts and turn hardcoded extra lines into pur paragraph spacing, is also an option of course. I don’t argue against that. But it also means that  the users of all these site now need to get used to this and stop making extra lines like they might have done for years.