Social Search

[All Astronauts]

Ugh... Not a bug, thought that was what was wanted. Which makes sense as excluding admins, the people running the site, from adding searches into the pile that you want to look at to see what your *users* are doing makes that data better. Otherwise the people running the site might skew the results.

Make sense?

Outside that I would need to actually store if the searcher was an admin or not in the database and then on display show or not... seems a lot of trouble for a couple of people

[Millipede]

That is what I wanted, and precisely for the reasons you mention 

It's just that as there's a button/option to enable or disable searches made by admins, I thought I'd test it and it doesn't show admin searches regardless of whether that button is on or off. Or have I misunderstood what that button/option does?

[All Astronauts]

Copy paste bug on my part. Silently updated, re-download pls and test.

[Millipede]

You are super fast! Thanks, it works a treat now

[All Astronauts]

4.1 Released!

• Fix on word and tag statistics for Cyrillic language users.

Anyone not using Cyrillic languages can skip this completely. Just a patch release.

[All Astronauts]

4.2 Released!

• Nothing special and not a required upgrade. No new features, just some minor code tweaks. Should be compatible with 4.3.6 but if not just use the 4.1 version. This was just a general sweep to ensure 4.4 compatibility. 4.1 is fine on that so, no biggie.

[Chris15440]

Hi, I installed 4.1 on 4.3.6 and the Popular Searches widget isn’t displaying any searches, even after being active for nearly 24 hours now. Searches are being made on the forum and are showing in the Recent Searches widget, just no popular searches. Any ideas to get this working properly? Thanks

[All Astronauts]

I'd say kick the socialSearch task but both those widgets run off the same ten minute task. 

I'd remove the widget, refresh the page, and re-add, then refresh again. If that doesn't clear it I'd need to peek under the hood (PM what I need if so)

Otherwise these are working on dev and the handful of live sites I control.

[All Astronauts]

Social Search 4.3 Released!

• Forgot to account for horizontal widget position on Popular Terms Widget. Derp.

• Also adds missing widget description language string.

[All Astronauts]

Social Search 4.4 Released!

• Patches word-wrap and out-of-box structure errors on front end widgets.

[All Astronauts]

Social Search 4.6 Released!

• 4.3: Forgot to account for horizontal widget position on Popular Terms Widget. Derp. Also adds missing widget description language string.
• 4.4: Patches word-wrap, out of-box problems with front end widgets.
• 4.5: Adds ability to not record searches in languages not in the Latin family. Any other language users wanting filters like this hit me up.
• 4.6: Fix for CIC users with stored URLs, no longer tracks searches caused by just going through search results pagination.

Figured I'd just post all of this as there have been a ton of changes recently. 4.5 wasn't actually released here, was a dev build.

The Latin language filter is "tested" for the value of nothing is breaking and searches are being recorded. 99% sure it's fine but yell if not. And yes, this is pretty hard filter - if the language the search being made in is not in the Latin family, Social Search will just not record it. The user will still make their search and so on, as always, Social Search does not interfere in anyway with the actual search - I just grab a copy of it when it is being made.

More importantly, I finally caught something that was bugging me for awhile. Right in front of me of course, and this will affect any stats you've been looking at  - nothing I can do there, sorry bout that. Anyways... When you make a search, get on the search results page, and have pages of search results, yours truly here spaced the obvious fact that when you go to page 2, 3, 4, etc... of search results, those are ALL NEW SEARCHES - they just adjust the results returned by x-amount. So 4.6 now no longer records those "searches" made when a user goes to page 2, etc. in results. You'll see a lot less double and triple searches in a row in the front end widgets and ledger.

[All Astronauts]

Social Search 4.7 Released!

• 4.7: Patches up memberSync so on member merge/delete the stored search urls, if they contain author filters, are correctly patched as needed.

[Claudia999]

Hi @All Astronauts

thanks for your plugin! 

But I've questions 🙂 and an example:

What are the red marked tags with "forums/xxx"? We don't have such tags especially for forums.

And the blue marked tags.. we don't have these tags in our community. The "search result" respectively the tag page is empty.

 

[All Astronauts]

What language is your forum using mostly? I'll take a look on my end but this seems like a bug. (specifically the apps appended to the tag with the slash, the blue circled one would be someone actually trying to search for just that tag)

Also what version IPS and I might need to get acp access to track this down if this ends up being language related,  but let me work things here first; I'll PM you if I need it. 

Edited November 21, 2019 by All Astronauts

[Claudia999]

It‘s german - and IPS is in the latest  version 

[All Astronauts]

Just a heads up that Social Search 5.0 will be kicked out in the not too distant future.

@Claudia999's problem was primarily a search bot gone absolutely mad coupled with what I suspect was a brief period where Advanced Tags and Prefixes was broken from an IPS update and that left some malformed tags which were then crawled, stored, and now those search urls are being run again by the various bots.

To that end, I've filtered the bots as much as one can - that's using the IPS bot detection (which is pretty low key) coupled with a few hard checks for 'bot', 'spider', and 'spyder' in the user_agent as a final check prior to storing the search. Outside of that not a lot I can do.

A few other tweaks and pokes, maybe a new thing, maybe. Out when it's out.

[All Astronauts]

So, for the nine of you that bought this, 5.0 is gonna roll out on Friday.

@Claudia999's search crawler hitch got me rolling and once that started a whole lot of other stuff came out of it. This is best described as part one, I still have half a list of things left to do but this will, as per usual, never get released if I keep adding and adding - that plus the fact there are only nine of you using this plus I need to step away from this just to clear my head for a bit 🥴

We'll leave the rest for later on, but tomorrow? You'll enjoy the changelog 🍺🍕

[All Astronauts]

Okie dokie, tested out on a live site of mine, caught an upgrade hitch or two, and I think that's that. Update will be live in the Marketplace a little bit after this post, writing it up here first before doing the Marketplace bit.

Version 5 Released!

• Option to disable links for guests/crawlers on the Search Wall and front side widgets and search results page recent searches block. The searches still display but just as words, not followable links.
• All front side links on the Search Wall, widgets, and results page recent searches block now carry ref=nofollow tags not that any bad actor crawlers care about such things...
• When determining to save searches, anything that at least publicly claims it is a crawler should now get sweeped. This is over and beyond the built-in IPS bot flagging. User agents are swept for matches on bot, crawler, spider, spyder, and totally empty user agent strings.
• Searches are now storing ip addresses for all guests (and bots that get through) and temporarily for searches coming from members. 
• User agent strings are stored for all searches. Future stats based on browser type, desktop vs. mobile, now possible. Also useful for bot hunting!
• Geolocation now stored for all searches (minus any addresses returned). Viewing a search in the ACP search ledger will display a map of where the search was made from. FYI geo-geolocation is an IPS service that only works for active licenses.
• Tooltips active for the guest/member icons in the search ledger. Ledger also has been reformatted.
• Duplicate searches based on case-insensitive match on the EXACT term (and term alone) are now no longer stored if coming from the same ip address within the last 20 minutes.
• Need to wipe the ledger and start over again? There's a button for that!
• General code improvements, abstractions, template changes, new templates...
• NEW! My Recent Searches feature for members! Last five searches made available from user menu. Also includes running count of number of searches made by member. This is entirely browser side via cookies. That means it is locked to the user's browser and not the user's member account and results will vary between devices. Nothing tracked internally to tie searches back to members in the ACP.

Before discussing the above you should probably see that I'm likely at the limit for not storing member ids with searches directly and instead abstracting out to "search made by a member". There certainly is an end user case to be made for storing them - looking back at all of the searches you have made, and it would be available across all devices, for one. Of course, once that's opened up as a feature that means GDPR and other compliance hilarity including the ability to dump out a user's searches on demand, nulling them on demand, and so on and so forth. You'd have to update your privacy junk, etc. I'm not adverse to it per se but it is a bunch of work I'd rather not get into yet; also the creep factor depending on the type of site you run.

Let's talk about the bot stuff first. As stated before, my guess is it was a bad actor crawler - the type that do not respect any noindex or nofollow tags. The search wall has always had a noindex output on it but, again, it requires the bot to give a damn about those things... The end result meant I had to make some changes. The nofollow tags on all the front end links is useful, but again, only useful for those bots following the rules. I guess if things are really bad for a site we need to flat out disable the links on the front side stuff for guests/bots. On the storing search side, still need to deal as stuff can still get through. For bot purposes IPS does flag out some bots, but it isn't that robust - that means I can use it as a front line check but need to do more. Alright, so lets start pulling user_agent strings into this and sweep them for some obvious things. I've got them so might as well store them and use that for search analysis as well later on (mobile vs. desktop, etc...). 

What about dupe searches? I could store member_ids but what about guests and bots? IP addresses it is then... And so on and etc... You can see once this door opened it just led on and on...

Geolocation was always on the road map and with the ip address and user_agent storing for bot hunting purposes it seemed a good time to add. The idea here is if you think you've got a bot sliding into the stored searches you can take a quick peak at the ledger, click one of the suspect searches, see the ip address and user agent string and see if it is skeezy or not. Check the ip address against known bot lists you can get online, or if running an active license, viewing search in the ledger will bring up the map and if it is a bot it will display your favorite India/Russia/China/SE Asia, Eastern Euro location 🥰 - Note to self, maybe button to spot check for bot in the search ledger? Hmmm... Geolocation has the potential to return an address - no worries, I'm not storing that.

This details view from the search ledger will get overhauled - bit rough as is, I know.

Naturally, with geolocation, there are some tasty stats/displays available in future versions of Social Search... Remember! Geolocation is an IPS service for ACTIVE licenses. Not active, no geo. That just means if it is not available when I store the searches, it doesn't get stored and won't be displayed. Will affect any stat stuff you do with geo in future versions if you drop in an out of active status though. Is what it is. Moving on...

IP address storage is forever (as long as you are storing searches) for guests/bots. That will of course include members who search while signed out (or people who are guests forever and then sign up) so there is a possibility to match stuff back to them if it's all from the same ip address. I'll think about a task to sweep for member-flagged searches and the IPS known devices stuff and clear on matches but thats a low low priority honestly. Not a concern for me really. Otherwise, on searches made by members, the id is still not stored, the search is just flagged as coming from a member and then the ip address is stored for a little while. Every six hours a task will sweep through and null out those ip addresses.

The dupe searches check kinda got pushed to the front with this bot stuff too - same search over and over again with really strange parameters appended. It can get really messy trying to match the exact parameters of a search being made when the intent is really to just clear out the person working through what is effectively the same search. So if a search comes in from the same ip address over the course of 20 minutes, case insensitive exact match on term, we'll not store it. I'll have that period of time configurable in a later version.

With all this going on, or maybe I just caught a similar thing in a side-glance while on a shopping site, I decided to let members see their last five searches. Since we are not storing member searches directly, that means device-based cookies. So if anyone asks, make sure they know it's only by device. If I ever move on to storing member ids with searches, it will be universal then, but not now.

    

Yes, that's a total search counter in the corner. Cookies are set to five years of life, like that will ever happen... And the large button should clearly indicate they can wipe this out and start over again anytime they like - its just the device cookie that clears, as, again, there are no specifically tied-to-the-member searches stored anywhere. I'll have settings for this in a later version - feature on/off (its on automatically now), group permissions so you can use it as a member perk, configurable amount of searches, different display options, etc. Early days here.

Version 6? I'll come back around to this soon enough but there is such a backlog of updates to other stuff... ack. Only so much we can do stats-wise internally, and although nothing is stopping you from dumping the database table and running with it I do have an export-to-csv feature on the list for those who want to do their own statistical analysis with the data with external sites/software.

If you are just reading this junk and thinking of buying, I'll be off the Black Friday 20% in a few hours but will carry on through to New Years at 15% off.

Merry Happy Joyous Whatevers.

EDIT: 115PM Central 12/6/2019 - it's up!

EDIT: Saturday Noon Central 12/7/2019 - small version bump to 5.0.2 5.0.3 to tackle small geolocation display hitch in search ledger. Sometimes geolocation service returns limited information such as only lat and long and I had some should-this-be-displayed checks set on another geo var. Now, if it provided city, region, country, those will display, and if it gave lat/long, the map should display (maps are generated based on lat/long) - and another one four hours later for My Recent Searches.

 

Edited December 7, 2019 by All Astronauts

[All Astronauts]

Whelp, that appears to be that. Not much incentive to carry this forward. Consider this app to be in maintenance mode from here on out. I imagine I will make sure its compatible with 4.5 whenever that hits, and bug fixes if needed of course, but any new features/dev would not be the best use of my time.

So it goes...

[Ember Stone]

Bought this plugin but was hoping to have something like this. The more term user search for it will go bigger and bolder in size accordingly.

[All Astronauts]

@Ember Stone In the widgets? I can do that.

[Ember Stone]

On search page on Recent Searches block

That will be nice if you can do here 

 

Edited April 23, 2020 by Ember Stone

[All Astronauts]

Might have to make some more adjusting there - those are literally the last x-number of searches made. Should not be a problem to provide an alternative most searched terms last 30/90 days thing there with the sized search terms.

[Ember Stone]

4 minutes ago, All Astronauts said:

Might have to make some more adjusting there - those are literally the last x-number of searches made. Should not be a problem to provide an alternative most searched terms last 30/90 days thing there with the sized search terms.

So any eta?

[All Astronauts]

Social Search 5.1 Released!

• Quick addition of a scaled text popular searches block to the results page - option in settings.
• Adjusted margins for searches displayed on Search Wall

@Ember Stone

This will NOT adjust the color of the results. Your site's colors are going to be so varied that it isn;t worth my time. You can adjust as you like by just adding the changes you need to your custom.css file.

For now, this will take the 20 most popular results and place them in this block (your mileage may vary based on how well-trafficked your community is + you will need tracked data so new installs maybe give it some time to get that data...). This block is at the moment only available on the search results page - it'll come to the front end widgets soon enough but my plate is full and so on on my end and with 4.5 looming (preview prior to beta is awfully close...) any additions to this will wait for my 4.5 sweep.

Oh yeah, the 20 are randomized on page load to mix it all up. (example above only has 14 - its my newer dev install with very little searching so not all the datas)

Popularity text sizing based on percentages - 10 steps - zero to nine. zero being the 'least most popular', nine being the 'most most popular'.

.ss-scale-0 {
	font-size: 1em;
}

.ss-scale-1 {
	font-size: 1.25em;
}

.ss-scale-2 {
	font-size: 1.5em;
}

.ss-scale-3 {
	font-size: 1.75em;
}

.ss-scale-4 {
	font-size: 2em;
}

.ss-scale-5 {
	font-size: 2em;
}

.ss-scale-6 {
	font-size: 2.25em;
}

.ss-scale-7 {
	font-size: 2.5em;
}

.ss-scale-8 {
	font-size: 2.75em;
}

.ss-scale-9 {
	font-size: 3em;
}

That's what I am providing. Copy/paste these into your custom.css and add in any color changes as needed to make the text darker or different colors as the percentages change. If you are not going to tweak the sizing then just remove those font-size bits to save on clutter.

To add this block, go to the Social Search settings and enable it. Note that the recent block and this new most popular block can both be on the page at the same time (as in the screen shot) - configure as you like.

Popularity is based on last 90 days.

Not bad for a request made a couple of hours ago 🍺

Edited April 23, 2020 by All Astronauts